Manpage for TXR

TXR is a general-purpose, multi-paradigm programming language. It comprises two languages integrated into a single tool: a text scanning and extraction language referred to as the TXR Pattern Language (sometimes just "TXR"), and a general-purpose dialect of Lisp called TXR Lisp.

TXR can be used for everything from "one liner" data transformation tasks at the command line, to data scanning and extracting scripts, to full application development in a wide range of areas.

A script written in the TXR Pattern Language, also referred to in this document as a query, specifies a pattern which matches one or more sources of inputs, such as text files. Patterns can consist of large chunks of multiline free-form text, which is matched literally against material in the input sources. Free variables occurring in the pattern (denoted by the @ symbol) are bound to the pieces of text occurring in the corresponding positions. Patterns can be arbitrarily complex, and can be broken down into named pattern functions, which may be mutually recursive.

In addition to embedded variables which implicitly match text, the TXR pattern language supports a number of directives, for matching text using regular expressions, for continuing a match in another file, for searching through a file for the place where an entire subquery matches, for collecting lists, and for combining subqueries using logical conjunction, disjunction and negation, and numerous others.

Patterns can contain actions which transform data and generate output. These actions can be embedded anywhere within the pattern-matching logic. A common structure for small TXR scripts is to perform a complete matching session at the top of the script, and then deal with processing and reporting at the bottom.

The TXR Lisp language can be used from within TXR scripts as an embedded language, or completely standalone. It supports functional, imperative and object-oriented programming, and provides numerous data types such as symbols, strings, vectors, hash tables with weak reference support, lazy lists, and arbitrary-precision ("bignum") integers. It has an expressive foreign function interface (FFI) for calling into libraries and other software components that support C-language-style calls.

TXR Lisp source files as well as individual functions can be optionally compiled for execution on a virtual machine that is built into TXR. Compiled files execute and load faster, and resist reverse-engineering. Standalone application delivery is possible.

TXR is free software offered under the two-clause BSD license which places almost no restrictions on redistribution, and allows every conceivable use, of the whole software or any constituent part, royalty-free, free of charge, and free of any restrictions.

4 ARGUMENTS AND OPTIONS

If TXR is given no arguments, it will enter into an interactive mode. See the INTERACTIVE LISTENER section for a description of this mode. When TXR enters interactive mode this way, it prints a one-line banner announcing the program name and version, and one line of help text instructing the user how to exit.

Unless the -c or -f options are present, the first non-option argument is treated as a script-file which is executed. This is described after the following descriptions of all of the options. Any additional arguments have no fixed meaning; they are available to the TXR query or TXR Lisp application for specifying input files to be processed, or other meanings under the control of the application.

Options which don't take an argument may be combined together. The -v and -q options are mutually exclusive. Of these two, the one which occurs in the rightmost position in the argument list dominates. The -c and -f options are also mutually exclusive; if both are specified, it is a fatal error.

If neither the -f nor the -c options were specified, then the first argument is treated as the script-file. If no arguments are present, then TXR enters interactive mode, provided that none of the -e, -p, -P or -t options had been processed, in which case it instead terminates.

The TXR Pattern Language has features for implicitly treating the subsequent command-line arguments as input files. It follows the convention that an argument consisting of a single - (dash) character specifies that standard input is to be used, instead of opening a file. If the query does not use the @(next) directive to select an alternative data source, and a pattern-matching construct is processed which demands data, then the first argument will be opened as a data source. Arguments not opened as data sources can be assigned alternative meanings and uses, or can be ignored entirely, under control of the query.

Specifying standard input as a source with an explicit - argument is unnecessary. If no arguments are present, then TXR scans standard input by default. This was not true in versions of TXR prior to 171; see the COMPATIBILITY section.

TXR begins by reading the script, which is given as the contents of the argument of the -c option, or else as the contents of an input source specified by the -f option or by the script-file argument. If -f or the script-file argument specify - (dash) then the script is read from standard input.

In the case of the TXR pattern language, the entire query is scanned, internalized, and then begins executing, if it is free of syntax errors. (TXR Lisp is processed differently, form by form.) On the other hand, the pattern language reads data files in a lazy manner. A file isn't opened until the query demands material from that file, and then the contents are read on demand, not all at once.

The suffix of the script-file is significant. If the name has no suffix, or if it has a ".txr" suffix, then it is assumed to be in the TXR pattern language. If it has the ".tl" suffix, then it is assumed to be TXR Lisp. The --lisp and --compiled options change the treatment of unsuffixed script file names, causing them to be interpreted as TXR Lisp source or compiled TXR Lisp, respectively.

If a file name is specified which does not have a recognized suffix, and names a file which doesn't exist, then TXR adds the ".txr" suffix and tries again. If that doesn't exist, another attempt is made with the ".tlo" suffix, which will be treated as as a TXR Lisp compiled file. Finally, if that doesn't exist, the ".tl" suffix is tried, which will be treated as containing TXR Lisp source. If either the --lisp or --compiled option has been specified, then TXR skips trying the ".txr" suffix, and tries only ".tlo" followed by ".tl".

A TXR Lisp file is processed as if by the load macro: forms from the file are read and evaluated. If the forms do not terminate the TXR process or throw an exception, and there are no syntax errors, then TXR terminates successfully after evaluating the last form. If syntax errors are encountered in a form, then TXR terminates unsuccessfully. TXR Lisp is documented in the section TXR LISP.

If a query file is specified, but no file arguments, it is up to the query to open a file, pipe or standard input via the @(next) directive prior to attempting to make a match. If a query attempts to match text, but has run out of files to process, the match fails.

5 STATUS AND ERROR REPORTING

TXR sends errors and verbose logs to the standard error device. The following paragraphs apply when TXR is run without enabling verbose mode with -v, or the printing of variable bindings with -B or -a.

If the command-line arguments are incorrect, TXR issues an error diagnostic and terminates with a failed status.

If the script-file specifies a query, and the query has a malformed syntax, TXR likewise issues error diagnostics and terminates with a failed status.

If the query fails due to a mismatch, TXR terminates with a failed status. No diagnostics are issued.

If the query is well-formed, and matches, then TXR issues no diagnostics, and terminates with a successful status.

In verbose mode (option -v), TXR issues diagnostics on the standard error device even in situations which are not erroneous.

In bindings-printing mode (options -B or -a), TXR prints the word false if the query fails, and exits with a failed termination status. If the query succeeds, the variable bindings, if any, are output on standard output.

If the script-file is TXR Lisp, then it is processed form by form. Each top-level Lisp form is evaluated after it is read. If any form is syntactically malformed, TXR issues diagnostics and terminates unsuccessfully. This is somewhat different from how the pattern language is treated: a script in the pattern language is parsed in its entirety before being executed.

6 BASIC TXR SYNTAX

6.1 Comments

A query may contain comments which are delimited by the sequence @; and extend to the end of the line. Whitespace can occur between the @ and ;. A comment which begins on a line swallows that entire line, as well as the newline which terminates it. In essence, the entire comment line disappears. If the comment follows some material in a line, then it does not consume the newline. Thus, the following two queries are equivalent:

The comment after the @a does not consume the newline, but the comment which follows does. Without this intuitive behavior, line comment would give rise to empty lines that must match empty lines in the data, leading to spurious mismatches.

Instead of the ; character, the # character can be used. This is an obsolescent feature.

6.2 Hash-Bang Support

TXR has several features which support use of the hash-bang convention for creating apparently standalone executable programs.

6.2.1 Basic Hash Bang

This removal allows for TXR queries to be turned into standalone executable programs in the POSIX environment using the hash-bang mechanism. Unlike most interpreters, TXR applies special processing to the #! line, which is described below, in the section Argument Generation with the Null Hack.

Shell session example: create a simple executable program called "twoline.txr" and run it. This assumes TXR is installed in /usr/bin.

When this plain hash-bang line is used, TXR receives the name of the script as an argument. Therefore, it is not possible to pass additional options to TXR. For instance, if the above script is invoked like this

the -B option isn't processed by TXR, but treated as an additional argument, just as if txrscript-file-B had been executed directly.

This behavior is useful if the script author wants not to expose the TXR options to the user of the script.

Now, the name of the script is passed as an argument to the -f option, and TXR will look for more options after that, so that the resulting program appears to accept TXR options. Now we can run

6.2.2 Argument Generation with --args and --eargs

To support systems like this, TXR supports the special argument --args, as well as an extended version, --eargs. With --args, it is possible to encode multiple arguments into one argument. The --args option must be followed by a separator character, chosen by the programmer. The characters after that are split into multiple arguments on the separator character. The --args option is then removed from the argument list and replaced with these arguments, which are processed in its place.

on a system which supports multiple arguments in the hash-bang line. The separator character is the colon, and so the remainder of that argument, -B:-f, is split into the two arguments -B -f.

The --eargs option is similar to --args, but must be followed by one more argument. After --eargs performs the argument splitting in the same manner as --args, any of the arguments which it produces which are the two-character sequence {} are replaced with that following argument. Whether or not the replacement occurs, that following argument is then removed.

This has an effect which cannot be replicated in any known implementation of the hash-bang mechanism. Suppose that this hash-bang line is placed in a script called script.txr. When this script is invoked with arguments, as in:

is produced by splitting into four fields using the : (colon) character as the separator. Then, within these four fields, all occurrences of {} are replaced with the following argument script.txr, resulting in:

Furthermore, that script.txr argument is removed from the remaining argument list.

The four arguments are then substituted in place of the original --eargs:-B:{}:--foo:42 syntax.

Thus, --eargs allows some arguments to be encoded into the interpreter script, such that script name is inserted anywhere among them, possibly multiple times. Arguments for the interpreter can be encoded, as well as arguments to be processed by the script.

6.2.3 Argument Generation with the Null Hack

Here, the env utility searches for the txr program in the directories indicated by the PATH variable, which liberates the script from having to encode the exact location where the program is installed. However, if the operating system allows only one argument in the hash-bang mechanism, then no arguments can be passed to the program.

To mitigate this problem, TXR supports a special feature in its hash-bang support. If the hash-bang line contains a null byte, then the text from after the null byte until the end of the line is split into fields using the space character as a separator, and these fields are inserted into the command line. This manipulation happens during command-line processing, i.e. prior to the execution of the file. If this processing is applied to a file that is specified using the -f option, then the arguments which arise from the special processing are inserted after that option and its argument. If this processing is applied to the file which is the first non-option argument, then the options are inserted before that argument. However, care is taken not to process that argument a second time. In either situation, processing of the command-line options continues, and the arguments which are processed next are the ones which were just inserted. This is true even if the options had been inserted as a result of processing the first non-option argument, which would ordinarily signal the termination of option processing.

In the following examples, it is assumed that the script is named, and invoked, as /home/jenny/foo.txr, and is given arguments --bar abc, and that txr resolves to /usr/bin/txr. The <NUL> code indicates a literal ASCII NUL character (the zero byte).

Here, env searches for txr, finding it in /usr/bin. Thus, including the executable name, TXR receives this full argument list:

The first non-option argument is the name of the script. TXR opens the script, and notices that it begins with a hash-bang line. It consumes the hash-bang line and finds the null byte inside it, retrieving the character string after it, which is "-a 3". This is split into the two arguments -a and 3, which are then inserted into the command line ahead of the the script name. The effective command line then becomes:

Command-line option processing continues, beginning with the -a option. After the option is processed, /home/jenny/foo.txr is encountered again. This time it is not opened a second time; it signals the end of option processing, exactly as it would immediately do if it hadn't triggered the insertion of any arguments.

Advanced example: use env to invoke txr, passing options to the interpreter and to the script:

This example shows how --eargs can be used in conjunction with the null hack. When txr begins executing, it receives the arguments

The script file is opened, and the arguments delimited by the null character in the hash-bang line are inserted, resulting in the effective command line:

Next, --eargs is processed in the ordinary way, transforming the command line into:

The name of the script file is encountered, and signals the end of option processing. Thus txr receives the -C option, instructing it to emulate some behaviors from version 175, and the /home/jenny/foo.txr script receives --debug as its argument: it executes with the *args* list containing one element, the character string "--debug".

The hash-bang null-hack feature was introduced in TXR 177. Previous versions ignore the hash-bang line, performing no special processing. Where a risk exists that programs which depend on the feature might be executed by an older version of TXR, care must be taken to detect and handle that situation, either by means of the txr-version variable, or else by some logic which infers that the processing of the hash-bang line hasn't been performed.

6.2.4 Passing Options to TXR via Hash-Bang Null Hack

It is possible to use the Hash-Bang Null Hack, such that the resulting executable program recognizes TXR options. This is made possible by a special behavior in the processing of the -f option.

For instance, suppose that the effect of the following familiar hash-bang line is required:

However, suppose there is also a requirement to use the env utility to find TXR. Furthermore, the operating system allows only one hash-bang argument. Using the Null Hack, this is rewritten as:

then if the script is invoked with arguments -a b c, the command line will ultimately be transformed into:

which allows TXR to process the -i option, leaving a, b and c as arguments for the script.

However, note that there is a subtle issue with the -f option that has been inserted via the Null Hack: namely, this insertion happens after TXR has opened the script file and read the hash-bang line from it. This means that when the inserted -f option is being processed, the script file is already open. A special behavior occurs. The -f option processing notices that the argument to -f is identical to the pathname of name of the script file that TXR has already opened for processing. The -f option and its argument are then skipped.

6.2.5 Hash Bang and Setuid

6.3 Whitespace

Outside of directives, whitespace is significant in TXR queries, and represents a pattern match for whitespace in the input. An extent of text consisting of an undivided mixture of tabs and spaces is a whitespace token.

Whitespace tokens match a precisely identical piece of whitespace in the input, with one exception: a whitespace token consisting of precisely one space has a special meaning. It is equivalent to the regular expression @/[ ]+/: match an extent of one or more spaces (but not tabs!). Multiple consecutive spaces do not have this meaning.

Thus, the query line "a b" (one space between a and b) matches "a b" with any number of spaces between the two letters.

For matching a single space, the syntax @\ can be used (backslash-escaped space).

It is more often necessary to match multiple spaces than to match exactly one space, so this rule simplifies many queries and inconveniences only a few.

In output clauses, string and character literals and quasiliterals, a space token denotes a space.

6.4 Text

Query material which is not escaped by the special character @ is literal text, which matches input character for character. Text which occurs at the beginning of a line matches the beginning of a line. Text which starts in the middle of a line, other than following a variable, must match exactly at the current position, where the previous match left off. Moreover, if the text is the last element in the line, its match is anchored to the end of the line.

An empty query line matches an empty line in the input. Note that an empty input stream does not contain any lines, and therefore is not matched by an empty line. An empty line in the input is represented by a newline character which is either the first character of the file, or follows a previous newline-terminated line.

Input streams which end without terminating their last line with a newline are tolerated, and are treated as if they had the terminator.

Text which follows a variable has special semantics, described in the section Variables below.

A query may not leave a line of input partially matched. If any portion of a line of input is matched, it must be entirely matched, otherwise a matching failure results. However, a query may leave unmatched lines. Matching only four lines of a ten-line file is not a matching failure. The eof directive can be used to explicitly match the end of a file.

In the following example, the query matches the text, even though the text has an extra line.

In the following example, the query fails to match the text, because the text has extra material on one line that is not matched:

Needless to say, if the text has insufficient material relative to the query, that is a failure also.

To match arbitrary material from the current position to the end of a line, the "match any sequence of characters, including empty" regular expression @/.*/ can be used. Example:

In this example, the query matches, since the regular expression matches the string "of data". (See the Regular Expressions section below.)

6.5 Special Characters in Text

Control characters may be embedded directly in a query (with the exception of newline characters). An alternative to embedding is to use escape syntax. The following escapes are supported:

Note that if a newline is embedded into a query line with @\n, this does not split the line into two; it's embedded into the line and thus cannot match anything. However, @\n may be useful in the @(cat) directive and in @(output).

6.6 Character Handling and International Characters

TXR represents text internally using wide characters, which are used to represent Unicode code points. Script source code, as well as all data sources, are assumed to be in the UTF-8 encoding. In TXR and TXR Lisp source, extended characters can be used directly in comments, literal text, string literals, quasiliterals and regular expressions. Extended characters can also be expressed indirectly using hexadecimal or octal escapes. On some platforms, wide characters may be restricted to 16 bits, so that TXR can only work with characters in the BMP (Basic Multilingual Plane) subset of Unicode.

TXR does not use the localization features of the system library; its handling of extended characters is not affected by environment variables like LANG and L_CTYPE. The program reads and writes only the UTF-8 encoding.

TXR deals with UTF-8 separately in its parser and in its I/O streams implementation.

TXR's text streams perform UTF-8 conversion internally, such that TXR applications use Unicode code points.

In text streams, invalid UTF-8 bytes are treated as follows. When an invalid byte is encountered in the middle of a multibyte character, or if the input ends in the middle of a multibyte character, or if an invalid character is decoded, such as an overlong from, or code in the range U+DC00 through U+DCFF, the UTF-8 decoder returns to the starting byte of the ill-formed multibyte character, and extracts just one byte, mapping that byte to the Unicode character range U+DC00 through U+DCFF, producing that code point as the decoded result. The decoder is then reset to its initial state and begins decoding at the following byte, where the same algorithm is repeated.

Furthermore, because TXR internally uses a null-terminated character representation of strings which easily interoperates with C language interfaces, when a null character is read from a stream, TXR converts it to the code U+DC00. On output, this code converts back to a null byte, as explained in the previous paragraph. By means of this representational trick, TXR can handle textual data containing null bytes.

In contrast to the above, the TXR parser scans raw UTF-8 bytes from a binary stream, rather than using a text stream. The parser performing its own recognition of UTF-8 sequences in certain language constructs, using a UTF-8 decoder only when processing certain kinds of tokens.

Comments are read without regard for encoding, so invalid encoding bytes in comments are not detected. A comment is simply a sequence of bytes terminated by a newline.

Invalid UTF-8 encountered while scanning identifiers and character names in character literal (hash-backslash) syntax is diagnosed as a syntax error.

UTF-8 in string literals is treated in the same way as UTF-8 in text streams. Invalid UTF-8 bytes are mapped into code points in the U+DC000 through U+DCFF range, and incorporated as such into the resulting string object which the literal denotes. The same remarks apply to regular-expression literals.

6.7 Regular Expression Directives

In place of a piece of text (see section Text above), a regular-expression directive may be used, which has the following syntax:

where the RE part enclosed in slashes represents regular-expression syntax (described in the section Regular Expressions below).

Long regular expressions can be broken into multiple lines using a backslash-newline sequence. Whitespace before the sequence or after the sequence is not significant, so the following two are equivalent:

Whereas literal text simply represents itself, regular expression denotes a (potentially infinite) set of texts. The regular-expression directive matches the longest piece of text (possibly empty) which belongs to the set denoted by the regular expression. The match is anchored to the current position; thus if the directive is the first element of a line, the match is anchored to the start of a line. If the regular-expression directive is the last element of a line, it is anchored to the end of the line also: the regular expression must match the text from the current position to the end of the line.

Even if the regular expression matches the empty string, the match will fail if the input is empty, or has run out of data. For instance suppose the third line of the query is the regular expression @/.*/, but the input is a file which has only two lines. This will fail: the data has no line for the regular expression to match. A line containing no characters is not the same thing as the absence of a line, even though both abstractions imply an absence of characters.

Like text which follows a variable, a regular-expression directive which follows a variable has special semantics, described in the section Variables below.

6.8 Variables

Much of the query syntax consists of arbitrary text, which matches file data character for character. Embedded within the query may be variables and directives which are introduced by a @ character. Two consecutive @@ characters encode a literal @.

A variable-matching or substitution directive is written in one of several ways:

@sident @{bident} @*sident @*{bident} @{bident /regex/} @{bident (fun [arg...])} @{bident number} @{bident bident}

The forms with an * indicate a long match, see Longest Match below. The forms with the embedded regexp /regex/ or function or number have special semantics; see Positive Match below.

The identifier t cannot be used as a name; it is a reserved symbol which denotes the value true. An attempt to use the variable @t will result in an exception. The symbol nil can be used where a variable name is required syntactically, but it has special semantics, described in a section below.

A sident consists of any combination of one or more letters, numbers, and underscores. It may not look like a number, so that for instance 123 is not a valid sident, but 12A is valid. Case is sensitive, so that FOO is different from foo, which is different from Foo.

The braces around an identifier can be used when material which follows would otherwise be interpreted as being part of the identifier. When a name is enclosed in braces it is a bident.

The following additional characters may be used as part of a bident which are not allowed in a sident:

Moreover, most Unicode characters beyond U+007F may appear in a bident, with certain exceptions. A character may not be used if it is any of the Unicode space characters, a member of the high or low surrogate region, a member of any Unicode private-use area, or is either of the two characters U+FFFE and U+FFFF. These situations produce a syntax error. Invalid UTF-8 in an identifier is also a syntax error.

The rule still holds that a name cannot look like a number so +123 is not a valid bident but these are valid: a->b, *xyz*, foo-bar.

The syntax @FOO_bar introduces the name FOO_bar, whereas @{FOO}_bar means the variable named "FOO" followed by the text "_bar". There may be whitespace between the @ and the name, or opening brace. Whitespace is also allowed in the interior of the braces. It is not significant.

If a variable has no prior binding, then it specifies a match. The match is determined from some current position in the data: the character which immediately follows all that has been matched previously. If a variable occurs at the start of a line, it matches some text at the start of the line. If it occurs at the end of a line, it matches everything from the current position to the end of the line.

6.9 Negative Match

then this is a "negative match". The extent of the matched text (the text bound to the variable) is determined by looking at what follows the variable, and ranges from the current position to some position where the following material finds a match. This is why this is called a "negative match": the spanned text which ends up bound to the variable is that in which the match for the trailing material did not occur.

A variable may be followed by a piece of text, a regular-expression directive, a function call, a directive, another variable, or nothing (i.e. occurs at the end of a line). These cases are described in detail below.

6.9.1 Variable Followed by Nothing

6.9.2 Variable Followed by Text

If a variable is followed by text, then the extent of the negative match is determined by searching for the first occurrence of that text within the line, starting at the current position.

The variable matches everything between the current position and the matching position (not including the matching position). Any whitespace which follows the variable (and is not enclosed inside braces that surround the variable name) is part of the text. For example:

In the above example, the pattern text "a b " matches the data "a b ". So when the @FOO variable is processed, the data being matched is the remaining "c d e f". The text which follows @FOO is " e f". This is found within the data "c d e f" at position 3 (counting from 0). So positions 0–2 ("c d") constitute the matching text which is bound to FOO.

6.9.3 Variable Followed by a Function Call or Directive

If the variable is followed by a function call, or a directive, the extent is determined by scanning the text for the first position where a match occurs for the entire remainder of the line. (For a description of functions, see Functions.)

Here, @foo will match the text from the current position to where "xyz" occurs, even though there is a @(bind) directive. Furthermore, if more material is added after the "xyz", it is part of the search. Note the difference between the following two:

In the first example, @foo matches the text from the current position until the match for the regular expression "abc". @(func) is not considered when processing @foo. In the second example, @foo matches the text from the current position until the position which matches the function call, followed by a match for the regular expression. The entire sequence @(func)@/abc/ is considered.

6.9.4 Consecutive Variables

However, what if an unbound variable with no modifier is followed by another variable? The behavior depends on the nature of the other variable.

If the other variable is also unbound, and also has no modifier, this is a semantic error which will cause the query to fail. A diagnostic message will be issued, unless operating in quiet mode via -q. The reason is that there is no way to bind two consecutive variables to an extent of text; this is an ambiguous situation, since there is no matching criterion for dividing the text between two variables. (In theory, a repetition of the same variable, like @FOO@FOO, could find a solution by dividing the match extent in half, which would work only in the case when it contains an even number of characters. This behavior seems to have dubious value.)

An unbound variable may be followed by one which is bound. The bound variable is effectively replaced by the text which it denotes, and the logic proceeds accordingly.

It is possible for a variable to be bound to a regular expression. If x is an unbound variable and y is bound to a regular expression RE, then @x@y means @x@/RE/. A variable v can be bound to a regular expression using, for example, @(bind v #/RE/).

Here, FOO is matched with "xyz", based on the delimiting around the colon. The colon in the pattern then matches the colon in the data, so that BAR is considered for matching against "defxyz". BAR is followed by FOO, which is already bound to "xyz". Thus "xyz" is located in the "defxyz" data following "def", and so BAR is bound to "def".

If an unbound variable is followed by a variable which is bound to a list, or nested list, then each character string in the list is tried in turn to produce a match. The first match is taken.

An unbound variable may be followed by another unbound variable which specifies a regular expression or function call match. This is a special case called a "double variable match". What happens is that the text is searched using the regular expression or function. If the search fails, then neither variable is bound: it is a matching failure. If the search succeeds, then the first variable is bound to the text which is skipped by the search. The second variable is bound to the text matched by the regular expression or function. Example:

6.9.5 Consecutive Variables via Directive

This is treated just like the variable followed by directive. No semantic error is identified, even if both variables are unbound. Here, @var2 matches everything at the current position, and so @var1 ends up bound to the empty string.

Example 2: *a specifies longest match (see Longest Match below), and so it takes everything:

6.9.6 Longest Match

In the former example, the match extends to the rightmost occurrence of "cd", and so FOO receives "b cdcdcd". In the latter example, the * syntax isn't used, and so a leftmost match takes place. The extent covers only the "b ", stopping at the first "cd" occurrence.

6.10 Positive Match

There are syntactic variants of variable syntax which have an embedded expression enclosed with the variable in braces:

@{bident /regex/} @{bident (fun [args...])} @{bident number} @{bident bident}

These specify a variable binding that is driven by a positive match derived from a regular expression, function or character count, rather than from trailing material (which is regarded as a "negative" match, since the variable is bound to material which is skipped in order to match the trailing material).

The positive match syntax is processed without considering any following syntax, and therefore may be followed by an unbound variable.

In the @{bident /regex/} form, the match extends over all characters from the current position which match the regular expression regex. (See the Regular Expressions section below.) If the variable already has a value, the text extracted by the regular expression must exactly match the variable.

In the @{bident (fun [args...])} form, the match extends over lines or characters which are matched by the call to the function, if the call succeeds. Thus @{x (y z w)} is just like @(y z w), except that the region of text skipped over by @(y z w) is also bound to the variable x. Except in one special case, the matching takes place horizontally within the current line, and the spanned range of text is treated as a string. The exception is that if the @{bident (fun [args...])} appears as the only element of a line, and fun has a binding as a vertical function, then the function is invoked in the same manner as it would be by the @(fun [args...]) syntax. Then the variable indicated by bident is bound to the list of lines matched by the function call. Pattern functions are described in the Functions section below. The function is invoked even if the variable already has a value. The text matched by the function must match the variable.

In the @{bident number} form, the match processes a field of text which consists of the specified number of characters, which must be a nonnegative number. If the data line doesn't have that many characters starting at the current position, the match fails. A match for zero characters produces an empty string. The text which is actually bound to the variable is all text within the specified field, but excluding leading and trailing whitespace. If the field contains only spaces, then an empty string is extracted. This fixed-field extraction takes place whether or not the variable already has a binding. If it already has a binding, then it must match the extracted, trimmed text.

The @{bident bident} syntax allows the number or regex modifier to come from a variable. The variable must be bound and contain a nonnegative integer or regular expression. For example, @{x y} behaves like @{x 3} if y is bound to the integer 3. It is an error if y is unbound.

6.11 Special Symbols nil and t

nil symbol stands for the empty list object, an object which marks the end of a list, and Boolean false. It is synonymous with the syntax () which may be used interchangeably with nil in most constructs.

In TXR Lisp, nil and t cannot be used as variables. When evaluated, they evaluate to themselves.

In the TXR pattern language, nil can be used in the variable binding syntax, but does not create a binding; it has a special meaning. It allows the variable-matching syntax to be used to skip material, in ways similar to the skip directive.

The nil symbol is also used as a block name, both in the TXR pattern language and in TXR Lisp. A block named nil is considered to be anonymous.

6.12 Keyword Symbols

Names beginning with the : (colon) character are keyword symbols. These also stand for themselves and may not be used as variables. Keywords are useful for labeling information and situations.

6.13 Regular Expressions

Regular expressions are a language for specifying sets of character strings. Through the use of pattern-matching elements, a regular expression is able to denote an infinite set of texts. TXR contains an original implementation of regular expressions, which supports the following syntax:

Any character which is not a regular-expression operator, a backslash escape, or the slash delimiter, denotes a one-position match of that character itself.

Any of the special characters, including the delimiting /, and the backslash, can be escaped with a backslash to suppress its meaning and denote the character itself.

Furthermore, all of the same escapes that are described in the section Special Characters in Text above are supported — the difference is that in regular expressions, the @ character is not required, so for example a tab is coded as \t rather than @\t. Octal and hex character escapes can be optionally terminated by a semicolon, which is useful if the following characters are octal or hex digits not intended to be part of the escape.

Only the above escapes are supported. Unlike in some other regular-expression implementations, if a backlash appears before a character which isn't a regex special character or one of the supported escape sequences, it is an error. This wasn't true of historic versions of TXR. See the COMPATIBILITY section.

The % operator is like a postfix operator with respect to its left operand, but like a unary operator with respect to its right operand. Thus a~b%c~d is a(~(b%(c(~d)))), demonstrating right-to-left associativity, where all of b% may be regarded as a unary operator being applied to c~d. Similarly, a?*+%b means (((a?)*)+)%b, where the trailing %b behaves like a postfix operator.

In TXR, regular expression matches do not span multiple lines. The regex language has no feature for multiline matching. However, the @(freeform) directive allows the remaining portion of the input to be treated as one string in which line terminators appear as explicit characters. Regular expressions may freely match through this sequence.

It's possible for a regular expression to match an empty string. For instance, if the next input character is z, facing the regular expression /a?/, there is a zero-character match: the regular expression's state machine can reach an acceptance state without consuming any characters. Examples:

In the first example, variable @A is followed by a regular expression which can match an empty string. The expression faces the letter z at position 0 in the data line. A zero-character match occurs there, therefore the variable A takes on the empty string. The @/.*/ regular expression then consumes the line.

Similarly, in the second example, the /a?/ regular expression faces a z, and thus yields an empty string which is bound to A. Variable @B consumes the entire line.

The third example requests the longest match for the variable binding. Thus, a search takes place for the rightmost position where the regular expression matches. The regular expression matches anywhere, including the empty string after the last character, which is the rightmost place. Thus variable A fetches the entire line.

For additional information about the advanced regular-expression operators, see NOTES ON EXOTIC REGULAR EXPRESSIONS below.

6.14 Compound Expressions

If the @ escape character is followed by an open parenthesis or square bracket, this is taken to be the start of a TXR Lisp compound expression.

The TXR language has the unusual property that its syntactic elements, so-called directives, are Lisp compound expressions. These expressions not only enclose syntax, but expressions which begin with certain symbols de facto behave as tokens in a phrase structure grammar. For instance, the expression @(collect) begins a block which must be terminated by the expression @(end), otherwise there is a syntax error. The collect expression can contain arguments which modify the behavior of the construct, for instance @(collect :gap 0 :vars (a b)). In some ways, this situation might be compared to HTML, in which an element such as <a> must be terminated by </a> and can have attributes such as <a href="...">.

Compound expressions contain subexpressions which are other compound expressions or literal objects of various kinds. Among these are: symbols, numbers, string literals, character literals, quasiliterals and regular expressions. These are described in the following sections. Additional kinds of literal objects exist, which are discussed in the TXR LISP section of the manual.

Symbols occurring in a compound expression follow a slightly more permissive lexical syntax than the bident in the syntax @{bident} introduced earlier. The / (slash) character may be part of an identifier, or even constitute an entire identifier. In fact a symbol inside a directive is a lident. This is described in the Symbol Tokens section under TXR LISP. A symbol must not be a number; tokens that look like numbers are treated as numbers and not symbols.

6.15 Character Literals

Character literals are introduced by the #\ (hash-backslash) syntax, which is either followed by a character name, the letter x followed by hex digits, the letter o followed by octal digits, or a single character. Valid character names are:

This convention for character literals is similar to that of the Scheme language. Note that #\linefeed and #\newline are the same character. The #\pnul character is specific to TXR and denotes the U+DC00 code in Unicode; the name stands for "pseudo-null", which is related to its special function. For more information about this, see the section "Character Handling and International Characters".

6.16 String Literals

String literals are delimited by double quotes. A double quote within a string literal is encoded using \" and a backslash is encoded as \\. Backslash escapes like \n and \t are recognized, as are hexadecimal escapes like \xFF or \xabc and octal escapes like \123. Ambiguity between an escape and subsequent text can be resolved by adding a semicolon delimiter after the escape: "\xabc;d" is a string consisting of the character U+0ABC followed by "d". The semicolon delimiter disappears. To write a literal semicolon immediately after a hex or octal escape, write two semicolons, the first of which will be interpreted as a delimiter. Thus, "\x21;;" represents "!;".

Note that the source code syntax of TXR string literals is specified in UTF-8, which is decoded into an internal string representation consisting of code points. The numeric escape sequences are an abstract syntax for specifying code points, not for specifying bytes to be inserted into the UTF-8 representation, even if they lie in the 8-bit range. Bytes cannot be directly specified, other than literally. However, when a TXR string object is encoded to UTF-8, every code point lying in the range U+DC00 through U+DCFF is converted to a single byte by taking the low-order eight bits of its value. By manipulating code points in this special range, TXR programs can reproduce arbitrary byte sequences in text streams. Also note that the \u escape sequence for specifying code points found in some languages is unnecessary and absent, since the existing hexadecimal and octal escapes satisfy this requirement. More detailed information is given in the earlier section Character Handling and International Characters.

If the line ends in the middle of a literal, it is an error, unless the last character is a backslash. This backslash is a special escape which does not denote a character; rather, it indicates that the string literal continues on the next line. The backslash is deleted, along with whitespace which immediately precedes it, as well as leading whitespace in the following line. The escape sequence "\ " (backslash space) can be used to encode a significant space.

The first string literal is the string "foobar". The second two are "foo bar".

6.17 Word List Literals

A word list literal (WLL) provides a convenient way to write a list of strings when such a list can be given as whitespace-delimited words.

There are two flavors of the WLL: the regular WLL which begins with #" (hash, double quote) and the splicing list literal which begins with #*" (hash, star, double quote).

Both types are terminated by a double quote, which may be escaped as \" in order to include it as a character. All the escaping conventions used in string literals can be used in word literals.

Unlike in string literals, whitespace (tabs and spaces) is not significant in word literals: it separates words. A whitespace character may be escaped with a backslash in order to include it as a literal character.

Just like in string literals, an unescaped newline character is not allowed. A newline preceded by a backslash is permitted. Such an escaped backslash, together with any leading and trailing unescaped whitespace, is removed and replaced with a single space.

A splicing word literal differs from a word literal in that it does not produce a list of string literals, but rather it produces a sequence of string literals that is merged into the surrounding syntax. Thus, the following two notations are equivalent:

The regular WLL produced a single list object, but the splicing WLL expanded into multiple string literal objects.

6.18 String Quasiliterals

Quasiliterals are similar to string literals, except that they may contain variable references denoted by the usual @ syntax. The quasiliteral represents a string formed by substituting the values of those variables into the literal template. If a is bound to "apple" and b to "banana", the quasiliteral `one @a and two @{b}s` represents the string "one apple and two bananas". A backquote escaped by a backslash represents itself. Unlike in directive syntax, two consecutive @ characters do not code for a literal @, but cause a syntax error. The reason for this is that compounding of the @ syntax is meaningful. Instead, there is a \@ escape for encoding a literal @ character. Quasiliterals support the full output variable syntax. Expressions within variable substitutions follow the evaluation rules of TXR Lisp. This hasn't always been the case: see the COMPATIBILITY section.

Quasiliterals can be split into multiple lines in the same way as ordinary string literals.

6.19 Quasiword List Literals

The quasiword list literals (QLLs) are to quasiliterals what WLLs are to ordinary literals. (See the above section Word List Literals.)

Just as in the case of WLLs, there are two flavors of the QLL: the regular QLL which begins with #` (hash, backquote) and the splicing QLL which begins with #*` (hash, star, backquote).

Both types are terminated by a backquote, which may be escaped as \` in order to include it as a character. All the escaping conventions used in quasiliterals can be used in QLLs.

Unlike in quasiliterals, whitespace (tabs and spaces) is not significant in QLLs: it separates words. A whitespace character may be escaped with a backslash in order to include it as a literal character.

A newline is not permitted unless escaped. An escaped newline works exactly the same way as it does in WLLs.

Note that the delimiting into words is done before the variable substitution. If the variable a contains spaces, then #`@a` nevertheless expands into a list of one item: the string derived from a.

A splicing QLL differs from an ordinary QLL in that it does not produce a list of quasiliterals, but rather it produces a sequence of quasiliterals that is merged into the surrounding syntax.

6.20 Numbers

An integer constant is made up of digits 0 through 9, optionally preceded by a + or - sign.

An integer constant can also be specified in hexadecimal using the prefix #x followed by an optional sign, followed by hexadecimal digits: 0 through 9 and the uppercase or lowercase letters A through F:

Similarly, octal numbers are supported with the prefix #o followed by octal digits:

A floating-point constant is marked by the inclusion of a decimal point, the scientific E notation, or both. It is an optional sign, followed by a mantissa consisting of digits, a decimal point, more digits, and then an optional E notation consisting of the letter e or E, an optional + or - sign, and then digits indicating the exponent value. In the mantissa, the digits are not optional. At least one digit must either precede the decimal point or follow it. That is to say, a decimal point by itself is not a floating-point constant.

In TXR there is a special "dotdot" token consisting of two consecutive periods. An integer constant followed immediately by dotdot is recognized as such; it is not treated as a floating constant followed by a dot. That is to say, 123.. does not mean 123. . (floating point 123.0 value followed by dot token). It means 123 .. (integer 123 followed by .. token).

Dialect Note: unlike in Common Lisp, 123. is not an integer, but the floating-point number 123.0.

6.21 Comments

Comments of the form @; were introduced earlier. Inside compound expressions, another convention for comments exists: Lisp comments, which are introduced by the ; (semicolon) character and span to the end of the line.

7 DIRECTIVES

7.1 Overview

When a TXR Lisp compound expression occurs in TXR preceded by a @, it is a directive.

Directives which are based on certain symbols are, additionally, involved in a phrase-structure syntax which uses Lisp expressions as if they were tokens.

not only denotes a compound expression with the collect symbol in its head position, but it also introduces a syntactic phrase which requires a matching @(end) directive. In other words, @(collect) is not only an expression, but serves as a kind of token in a higher-level, phrase-structure grammar.

Effectively, collect is a reserved symbol in the TXR language. A TXR program cannot use this symbol as the name of a pattern function due to its role in the syntax. The symbol has no reserved role in TXR Lisp.

Usually if this type of directive occurs alone in a line, not preceded or followed by other material, it is involved in a "vertical" (or line-oriented) syntax.

If such a directive is embedded in a line (has preceding or trailing material) then it is in a horizontal syntactic and semantic context (character-oriented).

Yet, this is considered one vertical item, which means that it does not match a line of data. (This is necessary because all horizontal syntax matches something within a line of data, which is undesirable for definitions.)

Many directives exhibit both horizontal and vertical syntax, with different but closely related semantics. Some are vertical only, some are horizontal only.

7.2 Subexpression Evaluation

Some directives contain subexpressions which are evaluated. Two distinct styles of evaluations occur in TXR: bind expressions and Lisp expressions. Which semantics applies to an expression depends on the syntactic context in which it occurs: which position in which directive.

The evaluation of TXR Lisp expressions is described in the TXR LISP section of the manual.

Bind expressions are so named because they occur in the @(bind) directive. TXR pattern function invocations also treat argument expressions as bind expressions.

The @(rebind), @(set), @(merge), and @(deffilter) directives also use bind expression evaluation. Bind expression evaluation also occurs in the argument position of the :tlist keyword in the @(next) directive.

Unlike Lisp expressions, bind expressions do not support operators. If a bind expression is a nested list structure, it is a template denoting that structure. Any symbol in any position of that structure is interpreted as a variable. When the bind expression is evaluated, those corresponding positions in the template are replaced by the values of the variables.

Anywhere where a variable can appear in a bind expression's nested list structure, a Lisp expression can appear preceded by the @ character. That Lisp expression is evaluated and its value is substituted into the bind expression's template.

Moreover, a Lisp expression preceded by @ can be used as an entire bind expression. The value of that Lisp expression is then taken as the bind expression value.

Any object in a bind expression which is not a nested list structure containing Lisp expressions or variables denotes itself literally.

7.3 Input Scanning and Data Manipulation

7.3.1 The next directive

The next directive indicates that the remaining directives in the current block are to be applied against a new input source.

It can only occur by itself as the only element in a query line, and takes various arguments, according to these possibilities:

@(next) @(nextsource) @(nextsource:nothrow) @(next :args) @(next :env) @(next :listlisp-expr) @(next :tlistbind-expr) @(next :stringlisp-expr) @(next :varvar) @(next nil)

The lone @(next) without arguments specifies that subsequent directives will match inside the next file in the argument list which was passed to TXR on the command line.

If source is given, it must be a TXR Lisp expression which denotes an input source. Its value may be a string or an input stream. For instance, if variable A contains the text "data", then @(next A) means switch to the file called "data", and @(next `@A.txt`) means to switch to the file "data.txt". The directive @(next (open-command `git log`)) switches to the input stream connected to the output of the git log command.

If the input source cannot be opened for whatever reason, TXR throws an exception (see Exceptions below). An unhandled exception will terminate the program. Often, such a drastic measure is inconvenient; if @(next) is invoked with the :nothrow keyword, then if the input source cannot be opened, the situation is treated as a simple match failure. The :nothrow keyword also ensures that when the stream is later closed, which occurs when the lazy list reads all of the available data, the implicit call to the close-stream function specifies nil as the argument value to that function's throw-on-error-p parameter. This :nothrow mechanism does not suppress all exceptions related to the processing of that stream; unusual conditions encountered during the reading of data from the stream may throw exceptions.

The variant @(next :args) means that the remaining command-line arguments are to be treated as a data source. For this purpose, each argument is considered to be a line of text. The argument list does include that argument which specifies the file that is currently being processed or was most recently processed. As the arguments are matched, they are consumed. This means that if a @(next) directive without arguments is executed in the scope of @(next :args), it opens the file named by the first unconsumed argument.

To process arguments, and then continue with the original file and argument list, wrap the argument processing in a @(block). When the block terminates, the input source and argument list are restored to what they were before the block.

The variant @(next :env) means that the list of process environment variables is treated as a source of data. It looks like a text file stream consisting of lines of the form "name=value". If this feature is not available on a given platform, an exception is thrown.

The syntax @(next :listlisp-expr) treats TXR Lisp expression lisp-expr as a source of text. The value of lisp-expr is flattened to a simple list in a way similar to the @(flatten) directive. The resulting list is treated as if it were the lines of a text file: each element of the list must be a string, which represents a line. If the strings happen contain embedded newline characters, they are a visible constituent of the line, and do not act as line separators.

The syntax @(next :tlistbind-expr) is similar to @(next :list ...) except that bind-expr is not a TXR Lisp expression, but a TXR bind expression.

The syntax @(next :varvar) requires var to be a previously bound variable. The value of the variable is retrieved and treated like a list, in the same manner as under @(next :list ...). Note that @(next :var x) is not always the same as @(next :tlist x), because :var x strictly requires x to be a TXR variable, whereas the x in :tlist x is an expression which can potentially refer to Lisp variable.

The syntax @(next :stringlisp-expr) treats expression lisp-expr as a source of text. The value of the expression must be a string. Newlines in the string are interpreted as line terminators.

binds a to "abc". Likewise, this is also the case with input files and other streams whose last line is not terminated by a newline.

However, watch out for empty strings, which are analogous to a correctly formed empty file which contains no lines:

This will not bind a to ""; it is a matching failure. The behavior of :list is different. The query

binds a to "". The reason is that under :list the string "" is flattened to the list ("") which is not an empty input stream, but a stream consisting of one empty line.

The @(next nil) variant indicates that the following subquery is applied to empty data, and the list of data sources from the command line is considered empty. This directive is useful in front of TXR code which doesn't process data sources from the command line, but takes command-line arguments. The @(next nil) incantation absolutely prevents TXR from trying to open the first command-line argument as a data source.

Note that the @(next) directive only redirects the source of input over the scope of subquery in which the that directive appears. For example, the following query looks for the line starting with "xyz" at the top of the file "foo.txt", within a some directive. After the @(end) which terminates the @(some), the "abc" is matched in the previous input stream which was in effect before the @(next) directive:

However, if the @(some) subquery successfully matched "xyz@suffix" within the file foo.text, there is now a binding for the suffix variable, which is visible to the remainder of the entire query. The variable bindings survive beyond the clause, but the data stream does not.

7.3.2 The skip directive

The skip directive considers the remainder of the query as a search pattern. The remainder is no longer required to strictly match at the current line in the current input stream. Rather, the current stream is searched, starting with the current line, for the first line where the entire remainder of the query will successfully match. If no such line is found, the skip directive fails. If a matching position is found, the remainder of the query is processed from that point.

The remainder of the query can itself contain skip directives. Each such directive performs a recursive subsearch.

Skip comes in vertical and horizontal flavors. For instance, skip and match the last line:

The skip directive has two optional arguments, which are evaluated as TXR Lisp expressions. If the first argument evaluates to an integer, its value limits the range of lines scanned for a match. Judicious use of this feature can improve the performance of queries.

Example: scan until "size: @SIZE" matches, which must happen within the next 15 lines:

Without the range limitation, skip will keep searching until it consumes the entire input source. In a horizontal skip, the range-limiting numeric argument is expressed in characters, so that

means: there must be a match for "abc" at the start of the line, and then within the next five characters, there must be a match for "def".

Sometimes a skip is nested within a collect, or following another skip. For instance, consider:

The above collect iterates over the entire input. But, potentially, so does the embedded skip. Suppose that "begin x" is matched, but the data has no matching "end x". The skip will search in vain all the way to the end of the data, and then the collect will try another iteration back at the beginning, just one line down from the original starting point. If it is a reasonable expectation that an end x occurs 15 lines of a "begin x", this can be specified instead:

If the symbol nil is used in place of a number, it means to scan an unlimited range of lines; thus, @(skip nil) is equivalent to @(skip).

If the symbol :greedy is used, it changes the semantics of the skip to longest match semantics. For instance, match the last three space-separated tokens of the line:

Without :greedy, the variable @c may match multiple tokens, and end up with spaces in it, because nothing follows @c and so it matches from any position which follows a space to the end of the line. Also note the space in front of @a. Without this space, @a will get an empty string.

A line-oriented example of greedy skip: match the last line without using @(eof):

There may be a second numeric argument. This specifies a minimum number of lines to skip before looking for a match. For instance, skip 15 lines and then search indefinitely for begin ...:

The two arguments may be used together. For instance, the following matches if and only if the 15th line of input starts with begin :

Essentially, @(skip 1n) means "hard skip by n lines". @(skip 1 0) is the same as @(skip 1), which is a noop, because it means: "the remainder of the query must match starting on the next line", or, more briefly, "skip exactly zero lines", which is the behavior if the skip directive is omitted altogether.

Non-greedy skip with the @(eof) directive has a slight advantage because the greedy skip will keep scanning even though it has found the correct match, then backtrack to the last good match once it runs out of data. The regular skip with explicit @(eof) will stop when the @(eof) matches.

7.3.3 Reducing Backtracking with Blocks

The skip directive can consume considerable CPU time when multiple skips are nested. Consider:

This is actually nesting: the second and third skips occur within the body of the first one, and thus this creates nested iteration. TXR is searching for the combination of skips which match the pattern of lines A, B and C with backtracking behavior. The outermost skip marches through the data until it finds A followed by a pattern match for the second skip. The second skip iterates to find B followed by the third skip, and the third skip iterates to find C. If A and B are only one line each, then this is reasonably fast. But suppose there are many lines matching A and B, giving rise to a large number of combinations of skips which match A and B, and yet do not find a match for C, triggering backtracking. The nested stepping which tries the combinations of A and B can give rise to a considerable running time.

One way to deal with the problem is to unravel the nesting with the help of blocks. For example:

Now the scope of each skip is just the remainder of the block in which it occurs. The first skip finds A, and then the block ends. Control passes to the next block, and backtracking will not take place to a block which completed (unless all these blocks are enclosed in some larger construct which backtracks, causing the blocks to be re-executed.

This rewrite is not equivalent, and cannot be used for instance in backreferencing situations such as:

7.3.4 The trailer directive

The trailer directive introduces a trailing portion of a query or subquery which matches input material normally, but in the event of a successful match, does not advance the current position. This can be used, for instance, to cause @(collect) to match partially overlapping regions.

A vertical trailer prevents the vertical input position from advancing as it is matched by directives, whereas a horizontal trailer prevents the horizontal position from advancing. In other words, trailer performs matching without consuming the input, providing a lookahead mechanism.

This script collects each line which has a duplicate somewhere later in the input. Without the @(trailer) directive, this does not work properly for inputs like:

Without @(trailer), the first duplicate pair constitutes a match which spans over the 222. After that pair is found, the matching continues after the second 111.

With the @(trailer) directive in place, the collect body, on each iteration, only consumes the lines matched prior to @(trailer).

7.3.5 The freeform directive

The freeform directive provides a useful alternative to TXR's line-oriented matching discipline. The freeform directive treats all remaining input from the current input source as one big line. The query line which immediately follows freeform is applied to that line.

where number and string denote TXR Lisp expressions which evaluate to an integer or string value, respectively.

If the number argument is given, its value limits the range of lines which are combined together. For instance @(freeform 5) means to only consider the next five lines to be one big line. Without this argument, freeform is "bottomless". It can match the entire file, which creates the risk of allocating a large amount of memory.

If the string argument is given, it specifies a custom line terminator. The default terminator is "\n". The terminator does not have to be one character long.

Freeform does not convert the entire remainder of the input into one big line all at once, but does so in a dynamic, lazy fashion, which takes place as the data is accessed. So at any time, only some prefix of the data exists as a flat line in which newlines are replaced by the terminator string, and the remainder of the data still remains as a list of lines.

After the subquery is applied to the virtual line, the unmatched remainder of that line is broken up into multiple lines again, by looking for and removing all occurrences of the terminator string within the flattened portion.

Care must be taken if the terminator is other than the default "\n". All occurrences of the terminator string are treated as line terminators in the flattened portion of the data, so extra line breaks may be introduced. Likewise, in the yet unflattened portion, no breaking takes place, even if the text contains occurrences of the terminator string. The extent of data which is flattened, and the amount of it which remains, depends entirely on the query line underneath @(flatten).

In the following example, lines of data are flattened using $ as the line terminator.

The data is turned into the virtual line 1$2:3$4$. The @a$@b: subquery matches the 1$2: portion, binding a to "1", and b to "2". The remaining portion 3$4$ is then split into separate lines again according to the line terminator $i:

faces these lines, binding c to 3 and d to 4. Note that since the data does not contain dollar signs, there is no ambiguity; the meaning may be understood in terms of the entire data being flattened and split again.

In the following example, freeform is used to solve a tokenizing problem. The Unix password file has fields separated by colons. Some fields may be empty. Using freeform, we can join the password file using ":" as a terminator. By restricting freeform to one line, we can obtain each line of the password file with a terminating ":", allowing for a simple tokenization, because now the fields are colon-terminated rather than colon-separated.

7.3.6 The fuzz directive

The fuzz directive allows for an imperfect match spanning a set number of lines. It takes two arguments, both of which are TXR Lisp expressions that should evaluate to integers:

This expresses that over the next n query lines, the matching strictness is relaxed a little bit. Only m out of those n lines have to match. Afterward, the rest of the query follows normal, strict processing.

In the degenerate situation where there are fewer than n query lines following the fuzz directive, then m of them must succeed anyway. (If there are fewer than m, then this is impossible.)

7.3.7 The line and chr directives

The line and chr directives perform binding between the current input line number or character position within a line, against an expression or variable:

The directive @(line 42) means "match the current input line number against the integer 42". If the current line is 42, then the directive matches, otherwise it fails. line is a vertical directive which doesn't consume a line of input. Thus, the following matches at the beginning of an input stream, and x ends up bound to the first line of input:

The directive @(line x) binds variable x to the current input line number, if x is an unbound variable. If x is already bound, then the value of x must match the current line number, otherwise the directive fails.

The chr directive is similar to line except that it's a horizontal directive, and matches the character position rather than the line position. Character positions are measured from zero, rather than one. chr does not consume a character. Hence the two occurrences of chr in the following example both match, and x takes the entire line of input:

The argument of line or chr may be an @-delimited Lisp expression. This is useful for matching computed lines or character positions:

7.3.8 The name directive

The name directive performs a binding between the name of the current data source and a variable or bind expression:

If na is an unbound variable, it is bound and takes on the name of the data source, such as a file name. If na is bound, then it has to match the name of the data source, otherwise the directive fails.

The directive @(name "data.txt") fails unless the current data source has that name.

7.3.9 The data directive

The data directive performs a binding between the unmatched data at the current position, and and a variable or bind expression. The unmatched data takes the form of a list of strings:

The binding is performed on object equality. If d is already bound, a matching failure occurs unless d contains the current unmatched data.

For instance, two branches of pattern matching can, at some point, bind the current data into different variables. When those paths join, the variables can be bound together to create the assertion that the current data had been the same at those points:

Here, two branches of the @(all) match some material which ends in the line bar. However, it is possible that this is a different line. The data directives are used to create an assertion that the data regions matched by the two branches are identical. That is to say, the unmatched data x captured after the first bar and the unmatched data y captured after the second bar must be the same object in order for @(require (eq x y)) to succeed, which implies that the same bar was matched in both branches of the @(all).

Another use of data is simply to gain access to the trailing remainder of the unmatched input in order to print it, or do some special processing on it.

The tprint Lisp function is useful for printing the unmatched data as newline-terminated lines:

7.3.10 The eof directive

The eof directive, if not given any argument, matches successfully when no more input is available from the current input source.

In the following example, the line variable captures the text "One-line file" and then since that is the last line of input, the eof directive matches:

The eof directive may be given a single argument, which is a pattern that matches the termination status of the input source. This is useful when the input source is a process pipe. For the purposes of eof, sources which are not process pipes have the symbol t as their termination status.

In the following example, which assumes the availability of a POSIX shell command interpreter in the host system, the variable a captures the string "a" and the status variable captures the integer value 5, which is the termination status of the command:

7.3.11 The some, all, none, maybe, cases and choose directives

These directives, called the parallel directives, combine multiple subqueries, which are applied at the same input position, rather than to consecutive input.

In horizontal mode, the current position is understood to be a character position in the line being processed. The clauses advance this character position by moving it to the right. In vertical mode, the current position is understood to be a line of text within the stream. A clause advances the position by some whole number of lines.

Long horizontal lines can be broken up with line continuations, allowing the above example to be written like this, which is considered a single logical line:

The @(some), @(all), @(none), @(maybe), @(cases) or @(choose) must be followed by at least one subquery clause, and be terminated by @(end). If there are two or more subqueries, these additional clauses are indicated by @(and) or @(or), which are interchangeable. The separator and terminator directives also must appear as the only element in a query line.

nesting can be indicated using whitespace between @ and the directive expression. Thus, the above is an @(all) query containing a @(skip) clause which applies to a @(some) that is followed by the text line "a dark". The @(some) clause combines the text line "it", and a @(none) clause which contains just one clause consisting of the line "was".

7.3.12 The require directive

The require directive evaluates a TXR Lisp expression. (See TXR LISP far below.) If the expression yields a true value, then it succeeds, and matching continues with the directives which follow. Otherwise the directive fails.

In the context of the require directive, the expression should not be introduced by the @ symbol; it is expected to be a Lisp expression.

7.3.13 The if directive

The if directive allows for conditional selection of pattern-matching clauses, based on the Boolean results of Lisp expressions.

A variant of the if directive is also available for use inside an output clauses, where it similarly allows for the conditional selection of output clauses.

@(iflisp-expr) . . . @(eliflisp-expr) . . . @(eliflisp-expr) . . . @(else) . . . @(end)

The @(elif) and @(else) clauses are all optional. If @(else) is present, it must be last, before @(end), after any @(elif) clauses. Any of the clauses may be empty.

More precisely, how the if directive works is as follows. The Lisp expressions are evaluated in order, starting with the if expression, then the elif expressions if any are present. If any Lisp expression yields a true result (any value other than nil) then evaluation of Lisp expressions stops. The corresponding clause of that Lisp expression is selected and pattern matching continues with that clause. The result of that clause (its success or failure, and any newly bound variables) is then taken as the result of the if directive. If none of the Lisp expressions yield true, and an else clause is present, then that clause is processed and its result determines the result of the if directive. If none of the Lisp expressions yield true, and there is no else clause, then the if directive is deemed to have trivially succeeded, allowing matching to continue with whatever directive follows it.

7.3.14 The Lisp if versus TXR if

The @(output) directive supports the embedding of Lisp expressions, whose values are interpolated into the output. In particular, Lisp if expressions are useful. For instance @(if expr "A" "B") reproduces A if expr yields a true value, otherwise B. Yet the @(if) directive is also supported in @(output). How the apparent conflict between the two is resolved is that the two take different numbers of arguments. An @(if) which has no arguments at all is a syntax error. One that has one argument is the head of the if directive syntax which must be terminated by @(end) and which takes the optional @(elif) and @(else) clauses. An @(if) which has two or more arguments is parsed as a self-contained Lisp expression.

7.3.15 The gather directive

Sometimes text is structured as items that can appear in an arbitrary order. When multiple matches need to be extracted, there is a combinatorial explosion of possible orders, making it impractical to write pattern matches for all the possible orders.

The gather directive is for these situations. It specifies multiple clauses which all have to match somewhere in the data, but in any order.

For further convenience, the lines of the first clause of the gather directive are implicitly treated as separate clauses.

The multiline clauses are optional. The gather directive takes keyword parameters, see below.

7.3.16 The until / last clause in gather

How gather works is that the text is searched for matches for the single-line and multiline queries. The clauses are applied in the order in which they appear. Whenever one of the clauses matches, any bindings it produces are retained and it is removed from further consideration. Multiple clauses can match at the same text position. The position advances by the longest match from among the clauses which matched. If no clauses match, the position advances by one line. The search stops when all clauses are eliminated, and then the cumulative bindings are produced. If the data runs out, but unmatched clauses remain, the directive fails.

Example: extract several environment variables, which do not appear in a particular order:

If the until or last clause is present and a match occurs, then the matches from the other clauses are discarded and the gather terminates. The difference between until/last is that any bindings bindings established in last are retained, and the input position is advanced past the matching material. The until/last clause has visibility to bindings established in the previous clauses in that same iteration, even though those bindings end up thrown away.

For consistency, the :mandatory keyword is supported in the until/last clause of gather. The semantics of using :mandatory in this situation is tricky. In particular, if it is in effect, and the gather terminates successfully by collecting all required matches, it will trigger a failure. On the other hand, if the until or last clause activates before all required matches are gathered, a failure also occurs, whether or not the clause is :mandatory.

Meaningful use of :mandatory requires that the gather be open-ended; it must allow some (or all) variables not to be required. The presence of the option means that for gather to succeed, all required variables must be gathered first, but then termination must be achieved via the until/last clause before all gather clauses are satisfied.

7.3.17 Keyword parameters in gather

Here, a, b and c are required variables, and d is optional, with the default value given by the Lisp expression "foo".

Firstly, even if all the clauses in the gather match successfully and are eliminated, the directive will fail if the required variables do not have bindings. It doesn't matter whether the bindings are existing, or whether they are established by gather.

Secondly, if some of the clauses of gather did not match, but all of the required variables have bindings, then the directive succeeds. Without the presence of :vars, it would fail in this situation.

Thirdly, if gather succeeds (all required variables have bindings), then all of the optional variables which do not have bindings are given bindings to their default values.

The expressions which give the default values are evaluated whenever the gather directive is evaluated, whether or not their values are used.

7.3.18 The collect directive

The repeat symbol may be specified instead of collect, which changes the meaning, see below:

The subquery is matched repeatedly, starting at the current line. If it fails to match, it is tried starting at the subsequent line. If it matches successfully, it is tried at the line following the entire extent of matched data, if there is one. Thus, the collected regions do not overlap. (Overlapping behavior can be obtained: see the @(trailer) directive.)

Unless certain keywords are specified, or unless the collection is explicitly failed with @(fail), it always succeeds, even if it collects nothing, and even if the until/last clause never finds a match.

If no until/last clause is specified, and the collect is not limited using parameters, the collection is unbounded: it consumes the entire data file.

7.3.19 The until / last clause in collect

If an until/last clause is specified, the collection stops when that clause matches at the current position.

If an until clause terminates collect, no bindings are collected at that position, even if the main clause matches at that position also. Moreover, the position is not advanced. The remainder of the query begins matching at that position.

If a last clause terminates collect, the behavior is different. Any bindings captured by the main clause are thrown away, just like with the until clause. However, the bindings in the last clause itself survive, and the position is advanced to skip over that material.

The line 42 is not collected, even though it matches @a. Furthermore, the @(until) does not advance the position, so variable c takes 42.

The 42 is not collected into a list, just like before. But now the binding captured by @b emerges. Furthermore, the position advances so variable now takes 6.

The binding variables within the clause of a collect are treated specially. The multiple matches for each variable are collected into lists, which then appear as array variables in the final output.

The query matches the data in three places, so each variable becomes a list of three elements, reported as an array.

Variables with list bindings may be referenced in a query. They denote a multiple match. The -D command-line option can establish a one-dimensional list binding.

The clauses of collect may be nested. Variable matches collated into lists in an inner collect are again collated into nested lists in the outer collect. Thus an unbound variable wrapped in N nestings of @(collect) will be an N-dimensional list. A one-dimensional list is a list of strings; a two-dimensional list is a list of lists of strings, etc.

It is important to note that the variables which are bound within the main clause of a collect, that is, the variables which are subject to collection, appear, within the collect, as normal one-value bindings. The collation into lists happens outside of the collect. So for instance in the query:

The left @x establishes a binding for some material preceding an equal sign. The right @x refers to that binding. The value of @x is different in each iteration, and these values are collected. What finally comes out of the collect clause is a single variable called x which holds a list containing each value that was ever instantiated under that name within the collect clause.

Also note that the until clause has visibility over the bindings established in the main clause. This is true even in the terminating case when the until clause matches, and the bindings of the main clause are discarded.

7.3.20 Keyword parameters in collect

Within the @(collect) syntax, it is possible to specify keyword parameters for additional control of the behavior. A keyword parameter consist of a keyword symbol followed by an argument, enclosed within the @(collect) syntax. The following are the supported keywords.

7.3.21 Specifying variables in collect

The :vars keyword allows the query writer to add discipline the collect body.

The argument to :vars is a list of variable specs. A variable spec is either a symbol, denoting a required variable, or a (symbol default-value) pair, where default-value is a Lisp expression whose value specifies a default value for the variable, which is optional.

When a :vars list is specified, it means that only the given variables can emerge from the successful collect. Any newly introduced bindings for other variables do not propagate. More precisely, whenever the collect body matches successfully, the following three rules apply:

In the event that collect does not match anything, the variables specified in :vars, whether required or optional, are all bound to empty lists. These bindings are established after the processing of the until/last clause, if present.

Here, if the body "@a @c" matches, an error will be thrown because one of the mandatory variables is b, and the body neglects to produce a binding for b.

Here, if "@a @b" matches, only a will be collected, but not b, because b is not in the variable list. Furthermore, because there is no binding for c in the body, a binding is created with the value "foo", exactly as if c matched such a piece of text.

In the following example, the assumption is that THIS NEVER MATCHES is not found anywhere in the input but the line THIS DOES MATCH is found and has a successor which is bound to a. Because the body did not match, the :vars a and b should be bound to empty lists. But a is bound by the last clause to some text, so this takes precedence. Only b is bound to an empty list.

The following means: do not allow any variables to propagate out of any iteration of the collect and therefore collect nothing:

Instead of writing @(collect :vars nil), it is possible to write @(repeat). @(repeat) takes all collect keywords, except for :vars. There is a @(repeat) directive used in @(output) clauses; that is a different directive.

7.3.22 Mandatory until and last

The until/last clause supports the option keyword :mandatory, exemplified by the following:

This means that the collect must be terminated by a match for the until/last clause, or else by an explicit @(accept).

Specifically, the collect cannot terminate due to simply running out of data, or exceeding a limit on the number of matches that may be collected. In those situations, if an until or last clause is present with :mandatory, the collect is deemed to have failed.

7.3.23 The coll directive

The coll directive is the horizontal version of collect. Whereas collect works with multiline clauses on line-oriented material, coll works within a single line. With coll, it is possible to recognize repeating regularities within a line and collect lists.

Here, the variable A is bound to tokens which match the regular expression /[^, ]+/: nonempty sequence of characters other than commas or spaces.

Like collect, coll searches for matches. If no match occurs at the current character position, it tries at the next character position. Whenever a match occurs, it continues at the character position which follows the last character of the match, if such a position exists.

If not bounded by an until clause, it will exhaust the entire line. If the until clause matches, then the collection stops at that position, and any bindings from that iteration are discarded. Like collect, coll also supports an until/last clause, which propagates variable bindings and advances the position. The :mandatory keyword is supported.

coll clauses nest, and variables bound within a coll are available to clauses within the rest of the coll clause, including the until/last clause, and appear as single values. The final list aggregation is only visible after the coll clause.

The behavior of coll leads to difficulties when a delimited variable are used to match material which is delimiter separated rather than terminated. For instance, entries in a comma-separated files usually do not appear as "a,b,c," but rather "a,b,c".

The 5 is missing because it isn't followed by a space, which the text-delimited variable match "@a " looks for. After matching "4 ", coll continues to look for matches, and doesn't find any. It is tempting to try to fix it like this:

The problem now is that the regular expression / ?/ (match either a space or nothing), matches at any position. So when it is used as a variable delimiter, it matches at the current position, which binds the empty string to the variable, the extent of the match being zero. In this situation, the coll directive proceeds character by character. The solution is to use positive matching: specify the regular expression which matches the item, rather than a trying to match whatever follows. The collect directive will recognize all items which match the regular expression:

The until clause can specify a pattern which, when recognized, terminates the collection. So for instance, suppose that the list of items may or may not be terminated by a semicolon. We must exclude the semicolon from being a valid character inside an item, and add an until clause which recognizes a semicolon:

Note that the @(end) is followed by a semicolon. That's because when the @(until) clause meets a match, the matching material is not consumed.

This repetition can be avoided by using @(last) instead of @(until) since @(last) consumes the terminating material.

Instead of the above regular-expression-based approach, this extraction problem can also be solved with cases:

7.3.24 Keyword parameters in coll

The shorthand @(rep) may be used instead of @(coll :vars nil). @(rep) takes all keywords, except :vars.

7.3.25 The flatten directive

The flatten directive can be used to convert variables to one-dimensional lists. Variables which have a scalar value are converted to lists containing that value. Variables which are multidimensional lists are flattened to one-dimensional lists.

7.3.26 The merge directive

destination is a variable, which receives a new binding. sources are bind expressions.

The merge directive provides a way of combining collected data from multiple nested lists in a way which normalizes different nesting levels among the sources. This directive is useful for combining the results from collects at different levels of nesting into a single nested list such that parallel elements are at equal depth.

A new binding is created for the destination variable, which holds the result of the operation.

The merge directive performs its special function if invoked with at least three arguments: a destination and two sources.

The one-argument case @(merge x) binds a new variable x and initializes it with the empty list and is thus equivalent to @(bind x). Likewise, the two-argument case @(merge x y) is equivalent to @(bind x y), establishing a binding for x which is initialized with the value of y.

To understand what merge does when two sources are given, as in @(merge C A B), we first have to define a property called depth. The depth of an atom such as a string is defined as 1. The depth of an empty list is 0. The depth of a nonempty list is one plus the depth of its deepest element. So for instance "foo" has depth 1, ("foo") has depth 2, and ("foo" ("bar")) has depth three.

We can now define a binary (two argument) merge(A, B) function as follows. First, merge(A, B) normalizes the values A and B to produce a pair of values which have equal depth, as defined above. If either value is an atom it is first converted to a one-element list containing that atom. After this step, both values are lists; and the only way an argument has depth zero is if it is an empty list. Next, if either value has a smaller depth than the other, it is wrapped in a list as many times as needed to give it equal depth. For instance if A is (a) and B is (((("b" "c") ("d" "e)))) then A is converted to (((("a")))). Finally, the list values are appended together to produce the merged result. In the case of the preceding two example values, the result is: (((("a"))) ((("b" "c") ("d" "e)))). The result is stored into a the newly bound destination variable C.

If more than two source arguments are given, these are merged by a left-associative reduction, which is to say that a three argument merge(X, Y, Z) is defined as merge(merge(X, Y), Z). The leftmost two values are merged, and then this result is merged with the third value, and so on.

7.3.27 The cat directive

The cat directive converts a list variable into a single piece of text. The syntax is:

The sep argument is a Lisp expression whose value specifies a separating piece of text. If it is omitted, then a single space is used as the separator.

7.3.28 The bind directive

The bind directive is a kind of pattern match, which matches one or more variables given in pattern against a value produced by the bind-expression on the right.

Variable names occurring in the pattern expression may refer to bound or unbound variables.

Binding occurs as follows. The tree structure of pattern and the value of bind-expression are considered to be parallel structures.

Any variables in pattern which are unbound receive a new binding, which is initialized with the structurally corresponding piece of the object produced by bind-expression.

Any variables in pattern which are already bound must match the corresponding part of the value of bind-expression, or else the bind directive fails. Variables which are already bound are not altered, retaining their current values even if the matching is inexact.

The simplest bind is of one variable against itself, for instance binding A against A:

This will throw an exception if A is not bound. If A is bound, it succeeds, since A matches itself.

Here, if A is unbound, it takes on the same value as B. If A is bound, it has to match B, or the bind fails. Matching means that either

The right-hand side does not have to be a variable. It may be some other object, like a string, quasiliteral, regexp, or list of strings, etc. For instance,

will bind the string "ab\tc" to the variable A if A is unbound. If A is bound, this will fail unless A already contains an identical string. However, the right-hand side of a bind cannot be an unbound variable, nor a complex expression that contains unbound variables.

The left-hand side of bind can be a nested list pattern containing variables. The last item of a list at any nesting level can be preceded by a . (dot), which means that the variable matches the rest of the list from that position.

7.3.29 Keywords in the bind directive

7.3.30 Lisp forms in the bind directive

TXR Lisp forms, introduced by @ may be used in the bind-expression argument of bind, or as the entire form. This is consistent with the rules for bind expressions.

Here, a is bound to the integer 4. The second bind then succeeds because the forms (+ 2 2) and (* 2 2) produce equal values.

7.3.31 The set directive

The set directive syntactically resembles bind, but is not a pattern match. It overwrites the previous values of variables with new values from the right-hand side. Each variable that is assigned must have an existing binding: set will not induce binding.

Destructuring assignment. A ends up with "A", B ends up with ("B1" "B2") and C binds to ("C1" "C2").

Note that set does not support a TXR Lisp expression on the left side, so the following are invalid syntax:

The second one is erroneous even though there is a variable on the left. Because it is preceded by the @ escape, it is a Lisp variable, and not a pattern variable.

The set directive also doesn't support Lisp expressions in the pattern, which must consist only of variables.

7.3.32 The rebind directive

The rebind directive resembles bind. It combines the semantics of local and bind into a single directive. The bind-expression is evaluated in the current environment, and its value remembered. Then a new environment is produced in which all the variables specified in pattern are absent. Then, the pattern is newly bound in that environment against the previously produced value, as if using bind.

The old environment with the previous variables is not modified; it continues to exist. This is in contrast with the set directive, which mutates existing bindings.

When the function terminates, the previous value of recursion-level is restored. The effect is less verbose and more efficient than the following equivalent

but it does not support any keyword arguments. The filtering features of bind do not make sense in rebind because the variables are always reintroduced into an environment in which they don't exist, whereas filtering applies in situations when bound variables are matched against values.

The rebind directive also doesn't support Lisp expressions in the pattern, which must consist only of variables.

7.3.33 The forget directive

Directives which follow the forget or local directive no longer see any bindings for the symbols mentioned in that directive, and can establish new bindings.

It is strongly recommended to use the @(local) spelling in functions, because the forgetting action simulates local variables: for the given symbols, the machine forgets any earlier variables from outside of the function, and consequently, any new bindings for those variables belong to the function. (Furthermore, functions suppress the propagation of variables that are not in their parameter list, so these locals will be automatically forgotten when the function terminates.)

7.3.34 The do directive

The do directive evaluates zero or more TXR Lisp expressions. (See TXR LISP far below.) The value of the expression is ignored, and matching continues with the directives which follow the do directive, if any.

In the context of the do directive, the expression should not be introduced by the @ symbol; it is expected to be a Lisp expression.

7.3.35 The mdo directive

Like the do directive, mdo (macro-time do) evaluates zero or more TXR Lisp expressions. Unlike do, mdo performs this evaluation immediately upon being parsed. Then it disappears from the syntax.

The effect of @(mdo e0 e1 e2 ...) is exactly like @(do (macro-time e0 e1 e2 ...)) except that do doesn't disappear from the syntax.

Another difference is that do can be used as a horizontal or vertical directive, whereas mdo is only vertical.

7.3.36 The in-package directive

The in-package directive shares the same syntax and semantics as the TXR Lisp macro of the same name:

The in-package directive is evaluated immediately upon being parsed, leaving no trace in the syntax tree of the surrounding TXR query.

It causes the *package* special variable to take on the package denoted by name.

The directive that name is either a string or symbol. An error exception is thrown if this isn't the case. Otherwise it searches for the package. If the package is not found, an error exception is thrown.

7.4 Blocks

7.4.1 Overview

Blocks are useful for terminating parts of a pattern-matching search prematurely, and escaping to a higher level. This makes blocks not only useful for simplifying the semantics of certain pattern matches, but also an optimization tool.

Judicious use of blocks and escapes can reduce or eliminate the amount of backtracking that TXR performs.

7.4.2 The block directive

The @(blockname) directive introduces a named block, except when name is the symbol nil. The @(block) directive introduces an unnamed block, equivalent to @(block nil).

The @(skip) and @(collect) directives introduce implicit anonymous blocks, as do function bodies.

7.4.3 Block Scope

The names of blocks are in a distinct namespace from the variable binding space. So @(block foo) is unrelated to the variable @foo.

A block extends from the @(block ...) directive which introduces it, until the matching @(end), and may be empty. For instance:

Here, the block foo occurs in a @(some) clause, and so it extends to the @(end) which terminates the block. After that @(end), the name foo is not associated with a block (is not "in scope"). The second @(end) terminates the @(some) block.

The implicit anonymous block introduced by @(skip) has the same scope as the @(skip): it extends over all of the material which follows the skip, to the end of the containing subquery.

7.4.4 Block Nesting

Blocks may nest, and nested blocks may have the same names as blocks in which they are nested. For instance:

is a nesting of two named blocks which happen to have the same name. When a nested block has the same name as an outer block, it creates a block scope in which the outer block is "shadowed"; that is to say, directives which refer to that block name within the nested block refer to the inner block, and not to the outer one.

7.4.5 Block Semantics

A block normally does nothing. The query material in the block is evaluated normally. However, a block serves as a termination point for @(fail) and @(accept) directives which are in scope of that block and refer to it.

At the point where the accept occurs, the foo block has matched the first line, bound the text "1" to the variable @first. The block is then terminated. Not only does the @first binding emerge from this terminated block, but what also emerges is that the block advanced the data past the first line to the second line. Next, the @(some) directive ends, and propagates the bindings and position. Thus the @second which follows then matches the second line and takes the text "2".

In the following query, the foo block occurs inside a maybe clause. Inside the foo block there is a @(some) clause. Its first subclause matches variable @first and then terminates block foo. Since block foo is outside of the @(some) directive, this has the effect of terminating the @(some) clause:

is never processed. The reason is that subclauses are processed in top to bottom order, but the processing was aborted within the first clause the @(accept foo). The @(some) construct never gets the opportunity to match four lines.

If the @(accept foo) line is removed from the above query, the output is different:

Now, all clauses of the @(some) directive have the opportunity to match. The second clause grabs four lines, which is the longest match. And so, the next line of input available for matching is 5, which goes to the @second variable.

7.4.6 Interaction Between the trailer and accept Directives

If one of the clauses which follow a @(trailer) requests a successful termination to an outer block via @(accept), then @(trailer) intercepts the escape and adjusts the data extent to the position that it was given.

The variable line3 is bound to "1" because although @(accept) yields a data position which has advanced to the third line, this is intercepted by @(trailer) and adjusted back to the first line. Neglecting to do this adjustment would violate the semantics of trailer.

7.4.7 Interaction Between the next and accept Directives

When the clauses under a next directive are terminated by an accept, such that control passes to a block which surrounds that next, the accept is intercepted by next.

The input position being communicated by the accept is replaced with the original input position in the original stream which is in effect prior to the next directive. The accept transfer is then resumed.

In other words, accept cannot be used to "leak" the new stream out of a next scope.

Here, the variable line matches the first line of the file "file-y", after which an accept transfer is initiated, targeting block b. This transfer communicates the line binding, as well as the position within file-y, pointing at the second line. However, the accept traverses the next directive, causing it to be abandoned. The special unwinding action within that directive detects this transfer and rewrites the input position to be the original one within the stream associated with "file-x". Note that this special handling exists in order for the behavior to be consistent with what would happen if the @(accept b) were removed, and the block b terminated normally: because the inner next is nested within that block, TXR would backtrack to the previous input position within "file-x".

7.4.8 Interaction Between Functions and the accept directive

@(define fun (a)) @ (bind a "a") @ (bind b "b") @ (accept blk) @(end) @(block blk) @(fun x) this line is skipped by accept @(end)

Here, the accept initiates a control transfer which communicates the a and b variable bindings which are visible in that scope. This transfer is intercepted by the function, and the treatment of the bindings follows to the same rules as a normal return (which, in the given function, would readily take place if the accept directive were removed). The b variable is suppressed, because b isn't a parameter of the function. Because a is a parameter, and the argument to that parameter is the unbound variable x, the effect is that x is bound to the value of a. When the accept transfer reaches block blk and terminates it, all that emerges is the x binding carrying "a".

If the accept invocation is removed from fun, then the function returns normally, producing the x binding. In that case, the line this line is skipped by accept isn't skipped since the block isn't being terminated; that line must match something.

7.4.9 Interaction Between finally and the accept directive

The processing of the finally block detects that it has been triggered by an accept transfer. Consequently, it retrieves the current input position and bindings from that transfer, and uses that position and those bindings for the processing of the finally clauses.

If the finally clauses succeed, then the new input position and new bindings are installed into the accept control transfer and that transfer resumes.

If the finally clauses fail, then the accept transfer is converted to a fail, with exactly the same block as its destination.

7.4.10 Vertical-Horizontal Mismatch Between block and accept

This creates the possibility that an accept in horizontal context targets a vertical block or vice versa, raising the question of how the input position is treated. The semantics of this is defined.

If a horizontal-context accept targets a vertical block, the current position at the target block will be the following line. That is to say, when the horizontal accept occurs, there is a current input line which may have unconsumed material past the current position. If the accept communicates its input position to a vertical context, that unconsumed material is skipped, as if it had been matched and the vertical position is advanced to the next line.

If a horizontal block catches a vertical accept, it rejects that accept's position and stays at the current backtracking position for that block. Only the bindings from the accept are retained.

7.4.11 Horizontal-Horizontal Mismatch between block and accept

7.5 Functions

7.5.1 Overview

Functions in TXR are not exactly like functions in mathematics or functional languages, and are not like procedures in imperative programming languages. They are not exactly like macros either. What it means for a TXR function to take arguments and produce a result is different from the conventional notion of a function.

A TXR function may have one or more parameters. When such a function is invoked, an argument must be specified for each parameter. However, a special behavior is at play here. Namely, some or all of the argument expressions may be unbound variables. In that case, the corresponding parameters behave like unbound variables also. Thus TXR function calls can transmit the "unbound" state from argument to parameter.

It should be mentioned that functions have access to all bindings that are visible in the caller; functions may refer to variables which are not mentioned in their parameter list.

With regard to returning, TXR functions are also unconventional. If the function fails, then the function call is considered to have failed. The function call behaves like a kind of match; if the function fails, then the call is like a failed match.

When a function call succeeds, then the bindings emanating from that function are processed specially. Firstly, any bindings for variables which do not correspond to one of the function's parameters are thrown away. Functions may internally bind arbitrary variables in order to get their job done, but only those variables which are named in the function argument list may propagate out of the function call. Thus, a function with no arguments can only indicate matching success or failure, but not produce any bindings. Secondly, variables do not propagate out of the function directly, but undergo a renaming. For each parameter which went into the function as an unbound variable (because its corresponding argument was an unbound variable), if that parameter now has a value, that value is bound onto the corresponding argument.

The above function collect-words contains a query which collects words from a line (sequences of characters other than space or tab), into the list variable called list. This variable is named in the parameter list of the function, therefore, its value, if it has one, is permitted to escape from the function call.

How it works is that in the function call @(collect-words wordlist), wordlist is an unbound variable. The parameter corresponding to that unbound variable is the parameter list. Therefore, that parameter is unbound over the body of the function. The function body collects the words of "Fine summer day" into the variable list, and then yields the that binding. Then the function call completes by noticing that the function parameter list now has a binding, and that the corresponding argument wordlist has no binding. The binding is thus transferred to the wordlist variable. After that, the bindings produced by the function are thrown away. The only enduring effects are:

Another way to understand the parameter behavior is that function parameters behave like proxies which represent their arguments. If an argument is an established value, such as a character string or bound variable, the parameter is a proxy for that value and behaves just like that value. If an argument is an unbound variable, the function parameter acts as a proxy representing that unbound variable. The effect of binding the proxy is that the variable becomes bound, an effect which is settled when the function goes out of scope.

Within the function, both the original variable and the proxy are visible simultaneously, and are independent. What if a function binds both of them? Suppose a function has a parameter called P, which is called with an argument A, which is an unbound variable, and then, in the function, both A and P bound. This is permitted, and they can even be bound to different values. However, when the function terminates, the local binding of A simply disappears (because the symbol A is not among the parameters of the function). Only the value bound to P emerges, and is bound to A, which still appears unbound at that point. The P binding disappears also, and the net effect is that A is now bound. The "proxy" binding of A through the parameter P "wins" the conflict with the direct binding.

7.5.2 Definition Syntax

Function definition syntax comes in two flavors: vertical and horizontal. Horizontal definitions actually come in two forms, the distinction between which is hardly noticeable, and the need for which is made clear below.

A function definition begins with a @(define ...) directive. For vertical functions, this is the only element in a line.

The define symbol must be followed by a symbol, which is the name of the function being defined. After the symbol, there is a parenthesized optional argument list. If there is no such list, or if the list is specified as () or the symbol nil then the function has no parameters. Examples of valid define syntax are:

If the define directive is followed by more material on the same line, then it defines a horizontal function:

If the define is the sole element in a line, then it is a vertical function, and the function definition continues below:

The difference between the two is that a horizontal function matches characters within a line, whereas a vertical function matches lines within a stream. The former match-x matches the character x, advancing to the next character position. The latter match-x matches a line consisting of the character x, advancing to the next line.

Material between @(define) and @(end) is the function body. The define directive may be followed directly by the @(end) directive, in which case the function has an empty body.

Functions may be nested within function bodies. Such local functions have dynamic scope. They are visible in the function body in which they are defined, and in any functions invoked from that body.

7.5.3 Two Forms of The Horizontal Function

If a horizontal function is defined as the only element of a line, it may not be followed by additional material. The following construct is erroneous:

This kind of definition is actually considered to be in the vertical context, and like other directives that have special effects and that do not match anything, it does not consume a line of input. If the above syntax were allowed, it would mean that the line would not only define a function but also match lalala. This would, in turn, would mean that the @(define)...@(end) is actually in horizontal mode, and so it matches a span of zero characters within a line (which means that is would require a line of input to match: a surprising behavior for a nonmatching directive!)

A horizontal function can be defined in an actual horizontal context. This occurs if its is in a line where it is preceded by other material. For instance:

This is a query line which must match the text XY. It also defines the function fun. The main use of this form is for nested horizontal functions:

7.5.4 Vertical-Horizontal Overloading

A function of the same name may be defined as both vertical and horizontal. Both functions are available at the same time. Which one is used by a call is resolved by context. See the section Vertical Versus Horizontal Calls below.

7.5.5 Call Syntax

A function is invoked by compound directive whose first symbol is the name of that function. Additional elements in the directive are the arguments. Arguments may be symbols, or other objects like string and character literals, quasiliterals ore regular expressions.

The first call to the function takes the line "one two". The parameter a takes "one" and parameter b takes "two". These are rebound to the arguments first and second. The second call to the function binds the a parameter to the word "ice", and the b is unbound, because the corresponding argument cream is unbound. Thus inside the function, a is forced to match ice. Then a space is matched and b collects the text "milk". When the function returns, the unbound "cream" variable gets this value.

If a symbol occurs multiple times in the argument list, it constrains both parameters to bind to the same value. That is to say, all parameters which, in the body of the function, bind a value, and which are all derived from the same argument symbol must bind to the same value. This is settled when the function terminates, not while it is matching. Example:

Here the query fails because a and b are effectively proxies for the same unbound variable same and are bound to different values, creating a conflict which constitutes a match failure.

7.5.6 Vertical Versus Horizontal Calls

A function call which is the only element of the query line in which it occurs is ambiguous. It can go either to a vertical function or to the horizontal one. If both are defined, then it goes to the vertical one.

Not only does this call go to the vertical function, but it is in a vertical context.

If only a horizontal function is defined, then that is the one which is called, even if the call is the only element in the line. This takes place in a horizontal character-matching context, which requires a line of input which can be traversed:

The query fails because since @(which fun) is in horizontal mode, it matches characters in a line. Since the function body consists only of @(bind ...) which doesn't match any characters, the function call requires an empty line to match. The line ABC is not empty, and so there is a matching failure. The following example corrects this:

A call made in a clearly horizontal context will prefer the horizontal function, and only fall back on the vertical one if the horizontal one doesn't exist. (In this fallback case, the vertical function is called with empty data; it is useful for calling vertical functions which process arguments and produce values.)

In the next example, the call is followed by trailing material, placing it in a horizontal context. Leading material will do the same thing:

7.5.7 Local Variables

As described earlier, variables bound in a function body which are not parameters of the function are discarded when the function returns. However, that, by itself, doesn't make these variables local, because pattern functions have visibility to all variables in their calling environment. If a variable x exists already when a function is called, then an attempt to bind it inside a function may result in a failure. The local directive must be used in a pattern function to list which variables are local.

This is a horizontal function which matches a path, which lands into four recursive cases. A path can be parenthesized path followed by a path; it can be a certain character followed by a path, or it can be empty

This function ensures that the variables it uses internally, x and y, do not have anything to do with any inherited bindings for x and y.

Note that the function is recursive, which cannot work without x and y being local, even if no such bindings exist prior to the top-level invocation of the function. The invocation @(path x) causes x to be bound, which is visible inside the invocation @(path y), but that invocation needs to have its own binding of x for local use.

7.5.8 Nested Functions

Function definitions may appear in a function. Such definitions are visible in all functions which are invoked from the body (and not necessarily enclosed in the body). In other words, the scope is dynamic, not lexical. Inner definitions shadow outer definitions. This means that a caller can redirect the function calls that take place in a callee, by defining local functions which capture the references.

Here, the function which is defined which calls fun. A top-level definition of fun is introduced which outputs "top-level fun!". The function callee provides its own local definition of fun which outputs "local fun!" before calling which. When callee is invoked, it calls which, whose @(fun) call is routed to callee's local definition. When which is called directly from the top level, its fun call goes to the top-level definition.

7.5.9 Indirect Calls

Function indirection may be performed using the call directive. If fun-expr is an Lisp expression which evaluates to a symbol, and that symbol names a function which takes no arguments, then

In this example, the effect is that foo is invoked, and b ends up bound to "abc".

The call directive here uses the 'foo expression to calculate the name of the function to be invoked. (See the quote operator).

This particular call expression can just be replaced by the direct invocation syntax @(foo b).

The power of call lies in being able to specify the function as a value which comes from elsewhere in the program, as in the following example.

Here the call directive obtains the name of the function from the f variable.

Note that function names are resolved to functions in the environment that is apparent at the point in execution where the call takes place. The directive @(call f args ...) is precisely equivalent to @(s args ...) if, at the point of the call, f is a variable which holds the symbol s and symbol s is defined as a function. Otherwise it is erroneous.

7.6 Modularization

7.6.1 The load and include directives

Where expr is a Lisp expression that evaluates to a string giving the path of the file to load.

Firstly, the path given by expr is converted to an effective path, as follows.

If the value of the *load-path* variable has a current value which is not nil and the path given in expr is pure relative according to the pure-rel-path-p function, then the effective path is interpreted taken relative to the directory portion of the path which is stored in *load-path*.

If *load-path* is nil, or the load path is not pure relative, then the path is taken as-is as the effective path.

Next, an attempt is made to open the file for processing, in almost exactly the same manner as by the TXR Lisp function load. The difference is that if the effective path is unsuffixed, then the .txr suffix is added to it, and that resulting path is tried first, and if it succeeds, then the file is treated as TXR Pattern Language syntax. If that fails, then the suffix .tlo is tried, and so forth, as described for the load function.

If these initial attempts to find the file fail, and the failure is due to the file not being found rather than some other problem such as a permission error, and expr isn't an absolute path according to abs-path-p, then additional attempts are made by searching for the file in the list of directories given in the *load-search-dirs* variable. Details are given in the description of the TXR Lisp load function.

Both the load and include directives bind the *load-path* variable to the path of the loaded file just before parsing syntax from it, The *package* variable is also given a new dynamic binding, whose value is the same as the existing binding. These bindings are removed when the load operation completes, restoring the prior values of these variables. The *load-hooks* variable is given a new dynamic binding, with a nil value.

If the file opened for processing is TXR Lisp source, or a compiled TXR Lisp file, then it is processed in the manner described for the load function.

Different requirements apply to the processing of the file under the load and include directives.

The include directive performs the processing of the file at parse time. If the file being processed is TXR Pattern Language, then it is parsed, and then its syntax replaces the include directive, as if it had originally appeared in its place. If a TXR Lisp source or a compiled TXR Lisp file is processed by include then the include directive is removed from the syntax.

The load directive performs the processing of the file at evaluation time. Evaluation time occurs after a TXR program is read from beginning to end and parsed. That is to say, when a TXR query is parsed, any embedded @(load ...) forms in it are parsed and constitute part of its syntax tree. They are executed when that query is executed, whenever its execution reaches those load directives. When the load directive processes TXR Pattern Language syntax, it parses the file in its entirety and then executes that file's directives against the current input position. Repeated executions of the same load directive result in repeated processing of the file.

Note: the include directive is useful for loading TXR files which contain Lisp macros which are needed by the parent program. The parent program cannot use load to bring in macros because macros are required during expansion, which takes place prior to evaluation time, whereas load doesn't execute until evaluation time.

7.7 Output

7.7.1 Introduction

A TXR query may perform custom output. Output is performed by output clauses, which may be embedded anywhere in the query, or placed at the end. Output occurs as a side effect of producing a part of a query which contains an @(output) directive, and is executed even if that part of the query ultimately fails to find a match. Thus output can be useful for debugging. An output clause specifies that its output goes to a file, pipe, or (by default) standard output. If any output clause is executed whose destination is standard output, TXR makes a note of this, and later, just prior to termination, suppresses the usual printing of the variable bindings or the word false.

7.7.2 The output directive

@(output [destination ] { bool-keyword | keyword value}* ) . . one or more output directives or lines . @(end)

If the directive has arguments, then the first one is evaluated. If it is an object other than a keyword symbol, then it specifies the optional destination. Any remaining arguments after the optional destination are the keyword list. If the destination is missing, then the entire argument list is a keyword list.

The destination argument, if present, is treated as a TXR Lisp expression and evaluated. The resulting value is taken as the output destination. The value may be a string which gives the pathname of a file to open for output. Otherwise, the destination must be a stream object.

The keyword list consists of a mixture of Boolean keywords which do not have an argument, or keywords with arguments.

7.7.3 Output Text

Text in an output clause is not matched against anything, but is output verbatim to the destination file, device or command pipe.

7.7.4 Output Variables

Variables occurring in an output clause do not match anything; instead their contents are output.

A variable being output can be any object. If it is of a type other than a list or string, it will be converted to a string as if by the tostring function in TXR Lisp.

A list is converted to a string in a special way: the elements are individually converted to a string and then they are catenated together. The default separator string is a single space: an alternate separation can be specified as an argument in the brace substitution syntax. Empty lists turn into an empty string.

Lists may be output within @(repeat) or @(rep) clauses. Each nesting of these constructs removes one level of nesting from the list variables that it contains.

In an output clause, the @{name number} variable syntax generates fixed-width field, which contains the variable's text. The absolute value of the number specifies the field width. For instance -20 and 20 both specify a field width of twenty. If the text is longer than the field, then it overflows the field. If the text is shorter than the field, then it is left-adjusted within that field, if the width is specified as a positive number, and right-adjusted if the width is specified as negative.

An output variable may specify a filter which overrides any filter established for the output clause. The syntax for this is @{NAME :filterfilterspec}. The filter specification syntax is the same as in the output clause. See Output Filtering below.

7.7.5 Output Variables: Indexing

Additional syntax is supported in output variables that does not appear in pattern-matching variables.

A square bracket index notation may be used to extract elements or ranges from a variable, which works with strings, vectors and lists. Elements are indexed from zero. This notation is only available in brace-enclosed syntax, and looks like this:

The above produces the text "b,c" in a field 10 spaces wide. The [1..3] argument extracts a range of a; the "," argument specifies an alternate separator string, and 10 specifies the field width.

7.7.6 Output Substitutions

The brace syntax has another syntactic and semantic extension in output clauses. In place of the symbol, an expression may appear. The value of that expression is substituted.

Here, the quasiliteral expression `@a:` is evaluated, producing the string "foo:". This string is printed right-adjusted in a 10 character field.

7.7.7 The repeat directive

The repeat directive generates repeated text from a "boilerplate", by taking successive elements from lists. The syntax of repeat is like this:

repeat has four types of special clauses, any of which may be specified with empty contents, or omitted entirely. They are described below.

All of the material in the main clause and optional clauses is examined for the presence of variables. If none of the variables hold lists which contain at least one item, then no output is performed, (unless the repeat specifies an @(empty) clause, see below). Otherwise, among those variables which contain nonempty lists, repeat finds the length of the longest list. This length of this list determines the number of repetitions, R.

If the repeat contains only a main clause, then the lines of this clause is output R times. Over the first repetition, all of the variables which, outside of the repeat, contain lists are locally rebound to just their first item. Over the second repetition, all of the list variables are bound to their second item, and so forth. Any variables which hold shorter lists than the longest list eventually end up with empty values over some repetitions.

Example: if the list A holds "1", "2" and "3"; the list B holds "A", "B"; and the variable C holds "X", then

will produce three repetitions (since there are two lists, the longest of which has three items). The output is:

The last line has a trailing space, since it is produced by "@A @B", where B has an empty value. Since C is not a list variable, it produces the same value in each repetition.

The precedence among the clauses which take an iteration is: single > first > modlast > last > mod > main. That is, whenever two or more of these clauses can apply to a repetition, then the leftmost one in this precedence list will be selected. It is possible for all these clauses to be viable for processing the same repetition. If a repeat occurs which has only one repetition, then that repetition is simultaneously the first, only and last repetition. Moreover, it also matches (mod 0 m) and, because it is the last repetition, it matches (modlast 0 m). In this situation, if there is a @(single) clause present, then the repetition shall be processed using that clause. Otherwise, if there is a @(first) clause present, that clause is activated. Failing that, @(modlast) is used if there is such a clause, featuring an n argument of zero. If there isn't, then the @(last) clause is considered, if present. Otherwise, the @(mod) clause is considered if present with an n argument of zero. Otherwise, none of these clauses are present or applicable, and the repetition is processed using the main clause.

The @(empty) clause does not appear in the above precedence list because it is mutually exclusive with respect to the others: it is processed only when there are no iterations, in which case even the main clause isn't active.

@(repeat [:counter {symbol | (symbol expr)}] [:vars ({symbol | (symbol expr)}*)])

The :counter argument designates a symbol which will behave as an integer variable over the scope of the clauses inside the repeat. The variable provides access to the repetition count, starting at zero, incrementing with each repetition. If the argument is given as (symbol expr) then expr is a Lisp expression whose value is taken as a displacement value which is added to each iteration of the counter. For instance :counter (c 1) specifies a counter c which counts from 1.

The :vars argument specifies a list of variable name symbols symbol or else pairs of the form (symbol init-form) consisting of a variable name and Lisp expression. Historically, the former syntax informed repeat about references to variables contained in Lisp code. This usage is no longer necessary as of TXR 243, since the repeat construct walks Lisp code, identifying all free variables. The latter syntax introduces a new pattern variable binding for symbol over the scope of the repeat construct. The init-form specifies a Lisp expression which is evaluated to produce the binding's value.

The repeat directive then processes the list of variables, selecting from it those which have a binding, either a previously existing binding or the one just introduced. For each selected variable, repeat will assume that the variable occurs in the repeat block and contains a list to be iterated.

The variable binding syntax supported by :vars of the form (symbol init-form) provides a solution for situations when it is necessary to iterate over some list, but that list is the result of an expression, and not stored in any variable. A repeat block iterates only over lists emanating from variables; it does not iterate over lists pulled from arbitrary expressions.

Example: output all file names matching the *.txr pattern in the current directory:

Prior to TXR 243, the simple variable-binding syntax supported by :vars of the form symbol was needed for situations in which TXR Lisp expressions which referenced variables were embedded in @(repeat) blocks. Variable references embedded in Lisp code were not identified in @(repeat). For instance, the following produced no output, because no variables were found in the repeat body:

There is a reference to trigraph but it's inside the (reverse trigraph) Lisp expression that was not processed by repeat. The solution was to mention trigraph in the :vars construct:

This workaround is no longer required as of TXR 243; the output is produced by the first example, without :vars.

7.7.8 Nested repeat directives

If a repeat clause encloses variables which hold multidimensional lists, those lists require additional nesting levels of repeat (or rep). It is an error to attempt to output a list variable which has not been decimated into primary elements via a repeat construct.

Suppose that a variable X is two-dimensional (contains a list of lists). X must be nested twice in a repeat. The outer repeat will traverse the lists contained in X. The inner repeat will traverse the elements of each of these lists.

A nested repeat may be embedded in any of the clauses of a repeat, not only in the main clause.

7.7.9 The rep directive

The rep directive is similar to repeat. Whereas repeat is line-oriented, rep generates material within a line. It has all the same clauses, but everything is specified within one line:

More than one @(rep) can occur within a line, mixed with other material. A @(rep) can be nested within a @(repeat) or within another @(rep).

7.7.10 repeat and rep Examples

Example 1: show the list L in parentheses, with spaces between the elements, or the word EMPTY if the list is empty:

Here, the @(empty) clause specifies EMPTY. So if there are no repetitions, the text EMPTY is produced. If there is a single item in the list L, then @(single)(@L) produces that item between parentheses. Otherwise if there are two or more items, the first item is produced with a leading parenthesis followed by a space by @(first)(@L and the last item is produced with a closing parenthesis: @(last)@L). All items in between are emitted with a trailing space by the main clause: @(rep)@L.

This is simpler. The parentheses are part of the text which surrounds the @(rep) construct, produced unconditionally. If the list L is empty, then @(rep) produces no output, resulting in (). If the list L has one or more items, then they are produced with spaces each one, except the last which has no space. If the list has exactly one item, then the @(last) applies to it instead of the main clause: it is produced with no trailing space.

7.7.11 The close directive

Where expr evaluates to a stream. The close directive can be used to explicitly close streams created using @(output ... :namedvar) syntax, as an alternative to @(output :finishexpr).

The same as above, using :finish rather than :continue so that the stream is closed at the end of the second block:

7.7.12 Output Filtering

Often it is necessary to transform the output to preserve its meaning under the convention of a given data format. For instance, if a piece of text contains the characters < or >, then if that text is being substituted into HTML, these should be replaced by < and >. This is what filtering is for. Filtering is applied to the contents of output variables, not to any template text. TXR implements named filters. Built-in filters are named by keywords, given below. User-defined filters are possible, however. See notes on the deffilter directive below.

Instead of a filter name, the syntax (funname) can be used. This denotes that the function called name is to be used as a filter. This is described in the next section Function Filters below.

To escape HTML characters in all variable substitutions occurring in an output clause, specify :filter :tohtml in the directive:

This will fold the contents of x to uppercase, and then encode any special characters into HTML. Beware of combinations that do not make sense. For instance, suppose the original text is HTML, containing codes like ". The compound filter (:upcase :fromhtml) will not work because " will turn to &QUOT; which no longer be recognized by the :fromhtml filter, since the entity names in HTML codes are case-sensitive.

7.7.13 Function Filters

A function can be used as a filter. For this to be possible, the function must conform to certain rules:

This function binds the out parameter to "bar" if the in parameter is "foo", otherwise it binds the out parameter to a copy of the in parameter. This is a simple filter.

To use the filter, use the syntax (:fun foo_to_bar) in place of a filter name. For instance in the bind directive:

The above should succeed since the left side is filtered from "foo" to "bar", so that there is a match.

Here is a split function which takes an extra argument which specifies the separator:

Furthermore, note that it produces a list rather than a string. This function separates the argument in into tokens according to the separator text carried in the variable sep.

When the filter invokes a function, it generates the first two arguments internally to pass in the input value and capture the output. The remaining arguments from the (:fun ...) construct are also passed to the function. Thus the string objects "," and "-" are passed as the sep argument to split and join.

Note that split puts out a list, which join accepts. So the overall filter chain operates on a string: a string goes into split, and a string comes out of join.

7.7.14 The deffilter directive

The deffilter directive allows a query to define a custom filter, which can then be used in output clauses to transform substituted data.

The deffilter symbol must be followed by the name of the filter to be defined, followed by bind expressions which evaluate to lists of strings. Each list must be at least two elements long and specifies one or more texts which are mapped to a replacement text. For instance, the following specifies a telephone keypad mapping from uppercase letters to digits.

The last deffilter has the same effect as the @(deffilter sub ("from" "to") ("---" "+++")) directive.

Filtering works using a longest match algorithm. The input is scanned from left to right, and the longest piece of text is identified at every character position which matches a string on the left-hand side, and that text is replaced with its associated replacement text. The scanning then continues at the first character after the matched text.

If none of the strings matches at a given character position, then that character is passed through the filter untranslated, and the scan continues at the next character in the input.

Filtering is not in-place but rather instantiates a new text, and so replacement text is not re-scanned for more replacements.

If a filter definition accidentally contains two or more repetitions of the same left-hand string with different right-hand translations, the later ones take precedence. No warning is issued.

7.7.15 The filter directive

A filter is specified, followed by one or more variables whose values are filtered and stored back into each variable.

7.8 Exceptions

7.8.1 Introduction

The exceptions mechanism in TXR is another disciplined form of nonlocal transfer, in addition to the blocks mechanism (see Blocks above). Like blocks, exceptions provide a construct which serves as the target for a dynamic exit. Both blocks and exceptions can be used to bail out of deep nesting when some condition occurs. However, exceptions provide more complexity. Exceptions are useful for error handling, and TXR in fact maps certain error situations to exception control transfers. However, exceptions are not inherently an error-handling mechanism; they are a structured dynamic control transfer mechanism, one of whose applications is error handling.

An exception control transfer (simply called an exception) is always identified by a symbol, which is its type. Types are organized in a subtype-supertype hierarchy. For instance, the file-error exception type is a subtype of the error type. This means that a file error is a kind of error. An exception handling block which catches exceptions of type error will catch exceptions of type file-error, but a block which catches file-error will not catch all exceptions of type error. A query-error is a kind of error, but not a kind of file-error. The symbol t is the supertype of every type: every exception type is considered to be a kind of t. (Mnemonic: t stands for type, as in any type).

In addition to being useful for exception handling, the @(try) directive also provides unwind protection by means of a @(finally) clause, which specifies query material to be executed unconditionally when the try clause terminates, no matter how it terminates.

7.8.2 The try directive

A try clause is surrounded by an implicit anonymous block (see Blocks section above). So for instance, the following is a no-op (an operation with no effect, other than successful execution):

The @(accept) causes a successful termination of the implicit anonymous block. Execution resumes with query lines or directives which follow, if any.

try clauses and blocks interact. For instance, an accept from within a try clause invokes a finally.

How this works: the try block's main clause is @(accept foo). This causes the enclosing block named foo to terminate, as a successful match. Since the try is nested within this block, it too must terminate in order for the block to terminate. But the try has a finally clause, which executes unconditionally, no matter how the try block terminates. The finally clause performs some output, which is seen.

Note that finally interacts with accept in subtle ways not revealed in this example; they are documented in the description of accept under the block directive documentation.

7.8.3 The finally clause

A try directive can terminate in one of three ways. The main clause may match successfully, and possibly yield some new variable bindings. The main clause may fail to match. Or the main clause may be terminated by a nonlocal control transfer, like an exception being thrown or a block return (like the block foo example in the previous section).

The finally clause is itself a query which binds variables, which leads to questions: what happens to such variables? What if the finally block fails as a query? As well as: what if a finally clause itself initiates a control transfer? Answers follow.

Firstly, a finally clause will contribute variable bindings only if the main clause terminates normally (either as a successful or failed match). If the main clause of the try block successfully matches, then the finally block continues matching at the next position in the data, and contributes bindings. If the main clause fails, then the finally block tries to match at the same position where the main clause failed.

The overall try directive succeeds as a match if either the main clause or the finally clause succeed. If both fail, then the try directive is a failed match.

In this example, the main clause of the try captures line "1" of the data as variable a, then the finally clause captures "2" as b, and then the query continues with the @c line after try block, so that c captures "3".

In this example, the main clause of the try fails to match, because the input is not prefixed with "hello ". However, the finally clause matches, binding b to "1". This means that the try block is a successful match, and so processing continues with @c which captures "2".

When finally clauses are processed during a nonlocal return, they have no externally visible effect if they do not bind variables. However, their execution makes itself known if they perform side effects, such as output.

A finally clause guards only the main clause and the catch clauses. It does not guard itself. Once the finally clause is executing, the try block is no longer guarded. This means if a nonlocal transfer, such as a block accept or exception, is initiated within the finally clause, it will not re-execute the finally clause. The finally clause is simply abandoned.

The disestablishment of blocks and try clauses is properly interleaved with the execution of finally clauses. This means that all surrounding exit points are visible in a finally clause, even if the finally clause is being invoked as part of a transfer to a distant exit point. The finally clause can make a control transfer to an exit point which is more near than the original one, thereby "hijacking" the control transfer. Also, the anonymous block established by the try directive is visible in the finally clause.

In this example, the @(next) directive throws an exception of type file-error, because the given file does not exist. The exit point for this exception is the @(catch file-error) clause in the outermost try block. The inner block is not eligible because it contains no catch clauses at all. However, the inner try block has a finally clause, and so during the processing of this exception which is headed for @(catch file-error), the finally clause performs an anonymous accept. The exit point for that accept is the anonymous block surrounding the inner try. So the original transfer to the catch clause is thereby abandoned. The inner try terminates successfully due to the accept, and since it constitutes the main clause of the outer try, that also terminates successfully. The "file error caught" message is never printed.

7.8.4 catch clauses

catch clauses establish their associated try blocks as potential exit points for exception-induced control transfers (called "throws").

A catch clause specifies an optional list of symbols which represent the exception types which it catches. The catch clause will catch exceptions which are a subtype of any one of those exception types.

If a try block has more than one catch clause which can match a given exception, the first one will be invoked.

When a catch is invoked, it is understood that the main clause did not terminate normally, and so the main clause could not have produced any bindings.

If a catch clause itself throws an exception, that exception cannot be caught by that same clause or its siblings in the same try block. The catch clauses of that block are no longer visible at that point. Nevertheless, the catch clauses are still protected by the finally block. If a catch clause throws, or otherwise terminates, the finally block is still processed.

If a finally block throws an exception, then it is simply aborted; the remaining directives in that block are not processed.

So the success or failure of the try block depends on the behavior of the catch clause or the finally clause, if there is one. If either of them succeed, then the try block is considered a successful match.

Here, the try block's main clause is terminated abruptly by a file-error exception from the @(next) directive. This is handled by the catch clause, which binds variable a to the input line "1". Then the finally clause executes, binding b to "2". The try block then terminates successfully, and so @c takes "3".

7.8.5 catch Clauses with Parameters

Parameters are useful in conjunction with throw. The built-in error exceptions carry one argument, which is a string containing the error message. Using throw, arbitrary parameters can be passed from the throw site to the catch site.

7.8.6 The throw directive

The throw directive generates an exception. A type must be specified, followed by optional arguments, which are bind expressions. For example,

throws an exception of type pair, with two arguments, being "a" and the expansion of the quasiliteral `@file.txt`.

The selection of the target catch is performed purely using the type name; the parameters are not involved in the selection.

Binding takes place between the arguments given in throw and the target catch.

If any catch parameter, for which a throw argument is given, is a bound variable, it has to be identical to the argument, otherwise the catch fails. (Control still passes to the catch, but the catch is a failed match).

If any argument is an unbound variable, the corresponding parameter in the catch is left alone: if it is an unbound variable, it remains unbound, and if it is bound, it stays as is.

If a catch has fewer parameters than there are throw arguments, the excess arguments are ignored:

If a catch has more parameters than there are throw arguments, the excess parameters are left alone. They may be bound or unbound variables.

A throw argument passing a value to a catch parameter which is unbound causes that parameter to be bound to that value.

throw arguments are evaluated in the context of the throw, and the bindings which are available there. Consideration of what parameters are bound is done in the context of the catch.

In the above example, c has a top-level binding to the string "c", but then becomes unbound via forget within the try construct, and rebound to the value "lc". Since the try construct is terminated by a throw, these modifications of the binding environment are discarded. Hence, at the end of the query, variable c ends up bound to the original value "c". The throw still takes place within the scope of the bindings set up by the try clause, so the values of a and c that are thrown are "a" and "lc". However, at the catch site, variable a does not have a binding. At that point, the binding to "a" established in the try has disappeared already. Being unbound, the catch parameter a can take whatever value the corresponding throw argument provides, so it ends up with "lc".

If throw is used to generate an exception derived from type error and that exception is not handled, TXR will issue diagnostics on the *stderr* stream and terminate. If an exception derived from warning is not handled, TXR will generate diagnostics on the *stderr* stream, after which control returns to the throw directive, and proceeds with the next directive. If an exception not derived from error is thrown, control returns to the throw directive and proceeds with the next directive.

7.8.7 The defex directive

The defex directive allows the query writer to invent custom exception types, which are arranged in a type hierarchy (meaning that some exception types are considered subtypes of other types).

Subtyping means that if an exception type B is a subtype of A, then every exception of type B is also considered to be of type A. So a catch for type A will also catch exceptions of type B. Every type is a supertype of itself: an A is a kind of A. This implies that every type is a subtype of itself also. Furthermore, every type is a subtype of the type t, which has no supertype other than itself. Type nil is a subtype of every type, including itself. The subtyping relationship is transitive also. If A is a subtype of B, and B is a subtype of C, then A is a subtype of C.

It may be invoked with one argument, which must be a symbol. This introduces a new exception type. Strictly speaking, such an introduction is not necessary; any symbol may be used as an exception type without being introduced by @(defex):

Therefore, this also does nothing, other than document the intent to use a as an exception.

If two or more argument symbols are given, the symbols are all introduced as types, engaged in a subtype-supertype relationship from left to right. That is to say, the first (leftmost) symbol is a subtype of the next one, which is a subtype of the next one and so on. The last symbol, if it had not been already defined as a subtype of some type, becomes a direct subtype of the master supertype t. Example:

The first directive defines d as a subtype of e, and e as a subtype of t. The second defines a as a subtype of b, b as a subtype of c, and c as a subtype of d, which is already defined as a subtype of e. Thus a is now a subtype of e. The above can be condensed to:

Exception types have a pervasive scope. Once a type relationship is introduced, it is visible everywhere. Moreover, the defex directive is destructive, meaning that the supertype of a type can be redefined. This is necessary so that something like the following works right:

These directives are evaluated in sequence. So after the first one, the ape type has the type t as its immediate supertype. But in the second directive, ape appears again, and is assigned the primate supertype, while retaining gorilla as a subtype. This situation could be diagnosed as an error, forcing the programmer to reorder the statements, but instead TXR obliges. However, there are limitations. It is an error to define a subtype-supertype relationship between two types if they are already connected by such a relationship, directly or transitively. So the following definitions are in error:

7.8.8 The assert directive

The assert directive requires the remaining query or subquery which follows it to match. If the remainder fails to match, the assert directive throws an exception. If the directive is simply

Then it throws an assertion of type assert, which is a subtype of error. The assert directive also takes arguments similar to the throw directive: an exception symbol and additional arguments which are bind expressions, and may be unbound variables. The following assert directive, if it triggers, will throw an exception of type foo, with arguments 1 and "2":

Without the assertion in places, if the Foo: @a, @b part does not match, then the entire interior of the @(collect) clause fails, and the collect continues searching for another match.

With the assertion in place, if the text "Important Header" and its underline match, then the remainder of the collect body must match, otherwise an exception is thrown. Now the program will not silently skip over any Important Header sections due to a problem in its matching logic. This is particularly useful when the matching is varied with numerous cases, and they must all be handled.

asserts that if the prefix "abc" is matched, then it must be followed by a successful match for "d@x", or else an exception is thrown.

If the exception is not handled, and is derived from error then TXR issues diagnostics on the *stderr* stream and terminates. If the exception is derived from warning and not handled, TXR issues a diagnostic on *stderr* after which control returns to the assert directive. Control silently returns to the assert directive if an exception of any other kind is not handled.

When control returns to assert due to an unhandled exception, it behaves like a failed match, similarly to the require directive.

8 TXR LISP

In any situation that calls for an expression, a Lisp expression can be used, if it is preceded by the @ character. The Lisp expression is evaluated and its value becomes the value of that expression. Thus, TXR directives are embedded in literal text using @, and Lisp expressions are embedded in directives using @ also.

Furthermore, certain directives evaluate Lisp expressions without requiring @. These are @(do), @(require), @(assert), @(if) and @(next).

TXR Lisp code can be placed into files. On the command line, TXR treats files with a ".tl" or ".tlo" suffix as TXR Lisp source or compiled code, and the @(load) directive does also.

Lastly, TXR Lisp expressions can be evaluated via the command line, using the -e and -p options.

Bind variable b to the standard input stream. Note that @ is not required on a Lisp variable:

8.1 Overview

TXR Lisp is a small and simple dialect, like Scheme, but much more similar to Common Lisp than Scheme. It has separate value and function binding namespaces, like Common Lisp (and thus is a Lisp-2 type dialect), and represents Boolean true and false with the symbols t and nil (note the case sensitivity of identifiers denoting symbols!). Furthermore, the symbol nil is also the empty list, which terminates nonempty lists.

TXR Lisp has lexically scoped local variables and dynamic global variables, similarly to Common Lisp, including the convention that defvar marks symbols for dynamic binding in local scopes. Lexical closures are supported. TXR Lisp also supports global lexical variables via defvarl.

Functions are lexically scoped in TXR Lisp; they can be defined in the pervasive global environment using defun or in local scopes using flet and labels.

8.2 Additional Syntax

Much of the TXR Lisp syntax has been introduced in the previous sections of the manual, since directive forms are based on it. There is some additional syntax that is useful in TXR Lisp programming.

8.2.1 Symbol Tokens

The symbol tokens in TXR Lisp, called a lident (Lisp identifier) has a similar syntax to the bident (braced identifier) in the TXR pattern language. It may consist of all the same characters, as well as the / (slash) character which may not be used in a bident. Thus a lident may consist of these characters, in addition to letters, numbers and underscores:

A lident may also include all of the Unicode characters which are permitted in a bident.

The one character which is allowed in a lident but not in a bident is / (forward slash).

A lone / is a valid lident and consequently a symbol token in TXR Lisp. The token /abc/ is also a symbol, and, unlike in a braced expression, is not a regular expression. In TXR Lisp expressions, regular expressions are written with a leading #.

8.2.2 Package Prefixes

If a symbol name contains a colon, the lident characters, if any, before that colon constitute the package prefix.

If the package exists, but the symbol name doesn't exist in that package, then the symbol is interned in that package.

If the package name is an empty string (the colon is preceded by nothing), the package is understood to be the keyword package. The symbol is interned in that package.

The syntax :test denotes the symbol test in the keyword package, the same as keyword:test.

Symbols in the keyword package are self-evaluating. This means that when a keyword symbol is evaluated as a form, the value of that form is the keyword symbol itself. Exactly two non-keyword symbols also have this special self-evaluating behavior: the symbols t and nil in the user package, whose fully qualified names are usr:t and usr:nil.

The syntax @foo:bar denotes the meta prefix @ being applied to the foo:bar symbol, not to a symbol in the @foo package.

The syntax #:bar denotes an uninterned symbol named bar, described in the next section.

8.2.3 Uninterned Symbols

Uninterned symbols are written with the #: prefix, followed by zero or more lident characters. When an uninterned symbol is read, a new, unique symbol is constructed, with the specified name. Even if two uninterned symbols have the same name, they are different objects. The make-sym and gensym functions produce uninterned symbols.

"Uninterned" means "not entered into a package". Interning refers to a process which combines package lookup with symbol creation, which ensures that multiple occurrences of a symbol name in written syntax are all converted to the same object: the first occurrence creates the symbol and associates it with its name in a package. Subsequent occurrences do not create a new symbol, but retrieve the existing one.

8.2.4 Meta-Atoms and Meta-Expressions

An expression may be preceded by the @ (at sign) character. If the expression is an atom, then this is a meta-atom, otherwise it is a meta-expression.

When the atom is a symbol, this is also called a meta-symbol and in situations when such a symbol behaves like a variable, it is also referred to as a meta-variable.

Meta-atom and meta-expression expressions have no evaluation semantics; evaluating them throws an exception. They play a syntactic role in the op operator, which makes use of meta-variables and meta-numbers, and in structural pattern matching, which uses meta-variables as pattern variables and whose operator vocabulary is based on meta-expressions.

In other situations, application code may assign meaning to meta syntax as the programmer sees fit.

If X is the syntax of an atom, such as a symbol, string or vector, then @X is a shorthand for the expression (sys:var X). Here, sys:var refers to the var symbol in the system-package.

If X is a compound expression, either (...) or [...], then @X is a shorthand for the expression (sys:expr X).

The behavior of @ followed by the syntax of a floating-point constant introduced by a leading decimal point, not preceded by digits, is unspecified. Examples of this are @.123 and @.123E+5.

The behavior of @ followed by the syntax of a floating-point expression in E notation, which lacks a decimal point, is also unspecified. An example of this is @12E5.

It is a syntax error for @ to be followed by what appears to be a floating-point constant consisting of a decimal point flanked by digits on both sides. For instance @1.2 is rejected.

A meta-expression followed by a period, and the syntax of another object is otherwise interpreted as a referencing dot expression. For instance @1.E3 denotes (qref @1 E3) which, in turn, denotes (qref (sys:var 1) E3), even though the unprefixed character sequence 1.E3 is otherwise a floating-point constant.

8.2.5 Consing Dot

Unlike other major Lisp dialects, TXR Lisp allows a consing dot with no forms preceding it. This construct simply denotes the form which follows the dot. That is to say, the parser implements the following transformation:

This is convenient in writing function argument lists that only take variable arguments. Instead of the syntax:

In no other circumstances is nil printed as (), or an atom sym as (. sym).

8.2.6 Referencing Dot

A dot token which is flanked by expressions on both sides, without any intervening whitespace, is the referencing dot, and not the consing dot. The referencing dot is a syntactic sugar which translated to the qref syntax ("quoted ref"). When evaluated as a form, this syntax denotes structure access; see Structures. However, it is possible to put this syntax to use for other purposes, in other contexts.

That is to say, this dot operator constructs a qref expression out of its left and right arguments. If the right argument of the dot is already a qref expression (whether produced by another instance of the dot operator, or expressed directly) it is merged. This requires the qref dot operator to be right-to-left associative, so that a.b.c works by first translating b.c to (qref b c), and then adjoining a to produce (qref a b c).

If the referencing dot is immediately followed by a question mark, it forms a single token, which produces the following syntactic variation, in which the following item is annotated as a list headed by the symbol t:

This syntax denotes null-safe access to structure slots and methods. a.?b means that a may evaluate to nil, in which case the expression yields nil; otherwise, a must evaluate to a struct which has a slot b, and the expression denotes access to that slot. Similarly, a.?(b 1) means that if a evaluates to nil, the expression yields nil; otherwise, a is treated as a struct object whose method b is invoked with argument 1, and the value returned by that method becomes the value of the expression.

Integer tokens cannot be involved in this syntax, because they form floating-point constants when juxtaposed with a dot. Such ambiguous uses of floating-point tokens are diagnosed as syntax errors:

8.2.7 Unbound Referencing Dot

Closely related to the referencing dot syntax is the unbound referencing dot. This is a dot which is flanked by an expression on the right, without any intervening whitespace, but is not preceded by an expression Rather, it is preceded by whitespace, or some punctuation such as [, ( or '. This is a syntactic sugar which translates to uref syntax:

If the unbound referencing dot is itself combined with a question mark to form the .? token, then the translation to uref is as follows:

When the unbound referencing dot is applied to a dotted expression, this can be understood as a conversion of qref to uref.

Indeed, this is exactly what happens if the unbound dot is applied to an explicit qref expression:

The unbound referencing dot takes its name from the semantics of the uref macro, which produces a function that implements late binding of an object to a method slot. Whereas the expression obj.a.b denotes accessing object obj to retrieve slot a and then accessing slot b of the object from that slot, the expression .a.b. represents a "disembodied" reference: it produces a function which takes an object as an argument and then performs the implied slot referencing on that argument. When the function is called, it is said to bind the referencing to the object. Hence that referencing is "unbound".

Whereas the expression .a produces a function whose argument must be an object, .?a produces a function whose argument may be nil. The function detects this case and returns nil.

8.2.8 Quote and Quasiquote

8.2.9 Quasiquoting non-List Objects

The #(1 2 3) literal is turned into a vector atom right in the TXR parser, and this atom is being quoted: this is (quoteatom) syntactically, which evaluates to atom.

When a vector is quasi-quoted, this is a case of ^atom which evaluates to atom.

In this situation, the ^#(...) notation produces code which constructs a vector.

The vector in the following example is also a quasivector. It contains unquotes, and though the quasiquote is not directly applied to it, it is embedded in a quasiquote:

Hash-table literals have two parts: the list of hash construction arguments and the key-value pairs. For instance:

where (:eql-based) indicates that this hash table's keys are treated using eql equality, and (a 1) and (b 2) are the key/value entries. Hash literals may be quasiquoted. In quasiquoting, the arguments and pairs are treated as separate syntax; it is not one big list. So the following is not a possible way to express the above hash:

8.2.10 Quasiquoting combined with Quasiliterals

8.2.11 Vector Literals

8.2.12 Struct Literals

8.2.13 Hash Literals

8.2.14 Range Literals

8.2.15 Buffer Literals

8.2.16 Tree Node Literals

8.2.17 Tree Literals

8.2.18 JSON Literals

8.2.19 The .. notation

That is to say, A .. B translates to (rcons A B), and so for instance (a b .. (c d) e .. f . g) means (a (rcons b (c d)) (rcons e f) . g).

The rcons function constructs a range object, which denotes a pair of values. Range objects are most commonly used for referencing subranges of sequences.

For instance, if L is a list, then [L 1 .. 3] computes a sublist of L consisting of elements 1 through 2 (counting from zero).

Note that if this notation is used in the dot position of an improper list, the transformation still applies. That is, the syntax (a . b .. c) is valid and produces the object (a . (rcons b c)) which is another way of writing (a rcons b c), which is quite probably nonsense.

The notation's .. operator associates right to left, so that a..b..c denotes (rcons a (rcons b c)).

Note that range objects are not printed using the dotdot notation. A range literal has the syntax of a two-element list, prefixed by #R. (See Range Literals above.)

In any context where the dotdot notation may be used, and where it is evaluated to its value, a range literal may also be specified. If an evaluated dotdot notation specifies two constant expressions, then an equivalent range literal can replace it. For instance the form [L 1 .. 3] can also be written [L #R(1 3)]. The two are syntactically different, and so if these expressions are being considered for their syntax rather than value, they are not the same.

8.2.20 The DWIM Brackets

For instance if foo is a variable which holds a function object, then [foo 3] can be used to call it, instead of (call foo 3). If foo is a vector, then [foo 3] retrieves the fourth element, like (vecref foo 3). Indexing over lists, strings and hash tables is possible, and the notation is assignable.

Furthermore, any arguments enclosed in [] which are symbols are treated according to a modified namespace lookup rule.

8.2.21 Compound Forms

The first position of an ordinary Lisp-2 style compound form, is expected to have a function or operator name. Then arguments follow. There may also be an expression in the dotted position, if the form is a function call.

If the form is a function call then the arguments are evaluated. If any of the arguments are symbols, they are treated according to Lisp-2 namespacing rules.

A function name may be a symbol, or else any of the syntactic forms given in the description of the function func-get-name.

8.2.22 Dot Position in Function Calls

If there is an expression in the dotted position of a function call expression, it is also evaluated, and the resulting value is involved in the function call in a special way.

Firstly, note that a compound form cannot be used in the dot position, for obvious reasons, namely that (a b c . (foo z)) does not mean that there is a compound form in the dot position, but denotes an alternate spelling for (a b c foo z), where foo behaves as a variable.

If the dot position of a compound form is an atom, then the behavior may be understood according to the following transformations:

In addition to atoms, meta-expressions and meta-symbols can appear in the dot position, even though their underlying syntax is comprised of a compound expression. This appears to work according to a transformation pattern which superficially appears to be the same as that for atoms:

However, in this situation, the @x is actually the form (sys:var x) and the dotted form is actually a proper list. The transformation is in fact taking place over a proper list, like this:

That is to say, the TXR Lisp form expander reacts to the presence of a sys:var or sys:expr atom in embedded in the form. That symbol and the items which follow it are wrapped in an additional level of nesting, converted into a single compound form element.

Effectively, in all these cases, the dot notation constitutes a shorthand for apply.

Note that the atom in the dot position of a function call may be a symbol macro. Since the semantics works as if by transformation to an apply form in which the original dot position atom is an ordinary argument, the symbol macro may produce a compound form.

That is to say, the expansion of x is not substituted into the form (list 1 . x) but rather the transformation to apply syntax takes place first, and so the substitution of x takes place in a form resembling (apply (fun list) 1 x).

In some other Lisp dialects like ANSI Common Lisp, the improper list syntax may not be used as a function call; a function called apply (or similar) must be used for application even if the expression which gives the trailing arguments is a symbol. Moreover, applying sequences other than lists is not supported.

8.2.23 Improper Lists as Macro Calls

TXR Lisp allows macros to be called using forms which are improper lists. These forms are simply destructured by the usual macro parameter list destructuring. To be callable this way, the macro must have an argument list which specifies a parameter match in the dot position. This dot position must either match the terminating atom of the improper list form, or else match the trailing portion of the improper list form.

then it may not be invoked as (mac 1 . 2) because the required argument b is not satisfied, and so the 2 argument cannot match the dot position c as required. The macro may be called as (mac 1 2 . 3) in which case c receives the form 3. If it is called as (mac 1 2 3 . 4) then c receives the improper list form 3 . 4.

8.2.24 Regular-Expression Literals

8.2.25 Notation for Circular and Shared Structure

TXR Lisp supports a printed notation called circle notation which accurately articulates the representation of objects which contain shared substructures as well as circular references. The notation is supported as a means of input, and is also optionally produced as output, controlled by the *print-circle* variable.

Ordinarily, shared substructure in printed objects is not evident, except in the case of multiple occurrences of interned symbols, in whose semantics it is implicit that they refer to the same object. Other shared structure is printed as separate copies which look like distinct objects. For instance, the object produced by (let ((shared '(1 2))) (list shared shared)) is printed as ((1 2) (1 2)), where it is not clear that the two occurrences of (1 2) are actually the same object. Under the circle notation, this object can be represented as (#5=(1 2) #5#). The #5= part introduces a reference label, associating the arbitrarily chosen nonnegative integer 5 with the object which follows. The subsequent notation #5# simply refers to the object labeled by 5, reproducing that object by reference. The result is a two-element list which has the same (1 2) in two places.

Circular structure presents a greater challenge to printing: namely, if it is printed by a naive recursive descent, it results in infinite output, and possibly stack exhaustion due to recursion. The circle notation detects and handles circular references. For instance, the object produced by (let ((c (list 1))) (rplacd c c)) produces a circular list which looks like an infinite list of 1's: (1 1 1 1 ...). This cannot be printed. However, under the circle notation, it can be represented as #1=(1 . #1#). The entire object itself is labeled by the integer 1. Then, enclosed within the syntax of that labeled object itself, a reference occurs to the label. This circular label reference represents the corresponding circular reference in the object.

Circular notation can span hash-table literals. The syntax #1=#H((:eql-based) (#1# #1#)) denotes an eql-based hash table which contains one entry, in which that same table itself is both the key and value. This kind of circularity is not supported for equal-based hash tables. The analogous syntax #1=#H(() (#1# #1#)) produces a hash table in an inconsistent state.

Circle notation is taken from Common Lisp, intended to be unsurprising to users familiar with that language. The implementation is based on descriptions in the ANSI Common Lisp document, judiciously taking into account the content of the X3J13 Cleanup Issues named PRINT-CIRCLE-STRUCTURE:USER-FUNCTIONS-WORK and PRINT-CIRCLE-SHARED:RESPECT-PRINT-CIRCLE.

8.2.26 Notation for Erasing Objects

Whereas it is valid for a TXR Lisp source file to be empty, it is a syntax error if a TXR Lisp source file contains nothing but one or more objects which are each suppressed by a preceding #;. In the interactive listener, an input line consisting of nothing but commented-out objects is similarly a syntax error.

The notation does not cascade; consecutive occurrences of #; trigger a syntax error.

The notation interacts with the circle notation. Firstly, if an object which is erased by #; contains circular-referencing instances of the label notation, those instances refer to nil. Secondly, commented-out objects may introduce labels which are subsequently referenced in expr. An example of the first situation occurs in:

Here the #1# label is a circular reference because it refers to an object which is a parent of the object which contains that reference. Such a reference is only satisfied by a "backpatching" process once the entire surrounding syntax is processed to the top level. The erasure perpetrated by #; causes the #1# label reference to be replaced by nil, and therefore the labeled object is the object (nil).

Here, even though the expression (#2=(a b c)) is suppressed, the label definition which it has introduced persists into the following object, where the label reference #2# resolves to (a b c).

which yields (nil). This is because the #1= label is available; but the earlier #1# reference, being a circular reference inside an erased object, had lapsed to nil.

8.3 Generalization of List Accessors

In ancient Lisp in the 1960's, it was not possible to apply the operations car and cdr to the nil symbol (empty list), because it is not a cons cell. In the InterLisp dialect, this restriction was lifted: these operations were extended to accept nil (and return nil). The convention was adopted in other Lisp dialects such as MacLisp and eventually in Common Lisp. Thus there exists an object which is not a cons, yet which takes car and cdr.

In TXR Lisp, this relaxation is extended further. For the sake of convenience, the operations car and cdr, are made to work with strings and vectors:

Moreover, structure types which define the methods car, cdr and nullify can also be treated in the same way.

The ldiff function is also extended in a special way. When the right parameter a non-list sequence, then it uses the equal equality test rather than eq for detecting the tail of the list.

The ldiff operation starts with "abcd" and repeatedly applies cdr to produce "bcd" and "cd", until the suffix is equal to the second argument: (equal "cd" "cd") yields true.

Operations based on car, cdr and ldiff, such as keep-if and remq extend to strings and vectors.

Most derived list processing operations such as remq or mapcar obey the following rule: the returned object follows the type of the leftmost input list object. For instance, if one or more sequences are processed by mapcar, and the leftmost one is a character string, the function is expected to return characters, which are converted to a character string. However, in the event that the objects produced cannot be assembled into that type of sequence, a list is returned instead.

For example [mapcar list "ab" "12"] returns ((#\a #\b) (#\1 #\2)), because a string cannot hold lists of characters. However [mappend list "ab" "12"] returns "a1b2".

The lazy versions of these functions such as mapcar* do not have this behavior; they produce lazy lists.

8.4 Generalization of Iteration

TXR Lisp implements a unified paradigm for iterating over sequence-like container structures and abstract spaces such as bounded and unbounded ranges of integers. This concept is based around an iterator abstraction which is directly compatible with Lisp cons-cell traversal in the sense that when iteration takes place over lists, the iterator instance is nothing but a cons cell.

An iterator is created using the constructor function iter-begin which takes a single argument. The argument denotes a space to be traversed; the iterator provides the means for that traversal.

When the iter-begin function is applied to a list (a cons cell or the nil object), the return value is that object itself. The remaining functions in the iterator API then behave like aliases for list processing functions. The iter-more function behaves like identity, iter-item behaves like car and iter-step behaves like cdr.

For example, the following loops not only produce identical behavior, but the iter variable steps through the cons cells in the same manner in both:

Firstly, both examples will still work if the list (a b c d) is replaced by a different kind of sequence, such as the string "abcd" or the vector #(a b c d). However, the former example will not execute efficiently on these objects. The reason is that the cdr function will construct successive suffixes of the string and list object. That requires not only the allocation of memory, but changes the running time complexity of the loop from linear to quadratic.

Secondly, the former example with car/cdr will not work correctly if the sequence is an empty non-list sequence, like the null string or empty vector. Rectifying this problem requires the nullify function to be used:

The nullify function converts empty sequences of all kinds into the empty list nil.

Thirdly, the second example will work even if the input list is replaced with certain objects which are not sequences at all:

In TXR Lisp, numerous functions that appear as list processing functions in other contemporary Lisp dialects, and historically, are actually sequence processing functions based on the above iterator paradigm.

8.5 Callable Objects

In TXR Lisp, sequences (strings, vectors and lists) as well as hashes and regular expressions can be used as functions everywhere, not just with the DWIM brackets.

Sequences work as one- or two-argument functions. With a single argument, an element is selected by position and returned. With two arguments, a range is extracted and returned.

Moreover, when a sequence is used as a function of one argument, and the argument is a range object rather than an integer, then the call is equivalent to the two-argument form. This is the basis for array slice syntax like ["abc" 0..1] .

Hashes also work as one or two argument functions, corresponding to the arguments of the gethash function.

A regular expression behaves as a one, two, or three argument function, which operates on a string argument. It returns the leftmost matching substring, or else nil.

Here, mapcar treats the string "abc" as a function of one argument (since there is one list argument). This function maps the indices 0, 1 and 2 to the corresponding characters of string "abc". Through this function, the list of integer indices (2 0 1) is taken to the list of characters (#\c #\a #\b).

Here, the shorthand 1 .. 3 denotes (rcons 1 3). A range used as an argument to a sequence performs range extraction: taking a slice starting at index 1, up to and not including index 3, as if by the call (sub '(1 2 3 4) 1 3).

A list of indices applied to a sequence is equivalent to using the select function, as if (select '(1 2 3 4) '(0 2)) were called.

Here, the regular expression, called as a function, finds the matching substring "bc" within the argument "abcd".

8.6 Special Variables

Similarly to Common Lisp, TXR Lisp is lexically scoped by default, but also has dynamically scoped (a.k.a "special") variables.

When a variable is defined with defvar or defparm, a binding for the symbol is introduced in the global name space, regardless of in what scope the defvar form occurs.

Furthermore, at the time the defvar form is evaluated, the symbol which names the variable is tagged as special.

When a symbol is tagged as special, it behaves differently when it is used in a lexical binding construct like let, and all other such constructs such as function parameter lists. Such a binding is not the usual lexical binding, but a "rebinding" of the global variable. Over the dynamic scope of the form, the global variable takes on the value given to it by the rebinding. When the form terminates, the prior value of the variable is restored. (This is true no matter how the form terminates; even if by an exception.)

Because of this "pervasive special" behavior of a symbol that has been used as the name of a global variable, a good practice is to make global variables have visually distinct names via the "earmuffs" convention: beginning and ending the name with an asterisk.

8.7 Syntactic Places and Accessors

The TXR Lisp feature known as syntactic places allows programs to use the syntax of a form which is used to access a value from an environment or object, as an expression which denotes a place where a value may be stored.

They are almost exactly the same concept as "generalized references" in Common Lisp, and are related to "lvalues" in languages in the C family, or "designators" in Pascal.

8.7.1 Symbolic Places

A symbol is a is a syntactic place if it names a variable. If a is a variable, then it may be assigned using the set operator: the form (set a 42) causes a to have the integer value 42.

8.7.2 Compound Places

A compound expression can be a syntactic place, if its leftmost constituent is as symbol which is specially registered, and if the form has the correct syntax for that kind of place, and suitable semantics. Such an expression is a compound place.

An example of a compound place is a car form. If c is an expression denoting a cons cell, then (car c) is not only an expression which retrieves the value of the car field of the cell. It is also a syntactic place which denotes that field as a storage location. Consequently, the expression (set (car c) "abc") stores the character string "abc" in that location. Although the same effect can be obtained with (rplaca c "abc") the syntactic place frees the programmer from having to remember different update functions for different kinds of places. There are various other advantages. TXR Lisp provides a plethora of operators for modifying a place in addition to set. Subject to certain usage restrictions, these operators work uniformly on all places. For instance, the expression (rotate (car x) [str 3] y) causes three different kinds of places to exchange contents, while the three expressions denoting those places are evaluated only once. New kinds of place update macros like rotate are quite easily defined, as are new kinds of compound places.

8.7.3 Accessor Functions

When a function call form such as the above (car x) is a syntactic place, then the function is called an accessor. This term is used throughout this document to denote functions which have associated syntactic places.

8.7.4 Macro Call Syntactic Places

Syntactic places can be macros (global and lexical), including symbol macros. So for instance in (set x 42) the x place can actually be a symbolic macro which expands to, say, (cdr y). This means that the assignment is effectively (set (cdr y) 42).

8.7.5 User-Defined Syntactic Places and Place Operators

Syntactic places, as well as operators upon syntactic places, are both open-ended. Code can be written quite easily in TXR Lisp to introduce new kinds of places, as well as new place-mutating operators. New places can be introduced with the help of the defplace, define-accessor or defset macros, or possibly the define-place-macro macro in simple cases when a new syntactic place can be expressed as a transformation to the syntax of an existing place. Three ways exist for developing new place update macros (place operators). They can be written using the ordinary macro definer ordinary macro definer defmacro, with the help of special utility macros called with-update-expander, with-clobber-expander, and with-delete-expander. They can also be written using defmacro in conjunction with the operators placelet or placelet*. Simple update macros similar to inc and push can be written compactly using define-modify-macro.

8.7.6 Deletable Places

Unlike generalized references in Common Lisp, TXR Lisp syntactic places support the concept of deletion. Some kinds of places can be deleted, which is an action distinct from (but does not preclude) being overwritten with a value. What exactly it means for a place to be deleted, or whether that is even permitted, depends on the kind of place. For instance a place which denotes a lexical variable may not be deleted, whereas a global variable may be. A place which denotes a hash-table entry may be deleted, and results in the entry being removed from the hash table. Deleting a place in a list causes the trailing items, if any, or else the terminating atom, to move in to close the gap. Users may define new kinds of places which support deletion semantics.

8.7.7 Evaluation of Places

To bring about their effect, place operators must evaluate one or more places. Moreover, some of them evaluate additional forms which are not places. Which arguments of a place operator form are places and which are ordinary forms depends on its specific syntax. For all the built-in place operators, the position of an argument in the syntax determines whether it is treated as (and consequently required to be) a syntactic place, or whether it is an ordinary form.

All built-in place operators perform the evaluation of place and non-place argument forms in strict left-to-right order.

Place forms are evaluated not in order to compute a value, but in order to determine the storage location. In addition to determining a storage location, the evaluation of a place form may possibly give rise to side effects. Once a place is fully evaluated, the storage location can then be accessed. Access to the storage location is not considered part of the evaluation of a place. To determine a storage location means to compute some hidden referential object which provides subsequent access to that location without the need for a reevaluation of the original place form. (The subsequent access to the place through this referential object may still require a multi-step traversal of a data structure; minimizing such steps is a matter of optimization.)

Place forms may themselves be compounds, which contain subexpressions that must be evaluated. All such evaluation for the built-in places takes place in left to right order.

Certain place operators, such as shift and rotate, exhibit an unspecified behavior with regard to the timing of the access of the prior value of a place, relative to the evaluation of places which occur later in the same place operator form. Access to the prior values may be delayed until the entire form is evaluated, or it may be interleaved into the evaluation of the form. For example, in the form (shift a b c 1), the prior value of a can be accessed and saved as soon as a is evaluated, prior to the evaluation of b. Alternatively, a may be accessed and saved later, after the evaluation of b or after the evaluation of all the forms. This issue affects the behavior of place-modifying forms whose subforms contain side effects. It is recommended that such forms not be used in programs.

8.7.8 Nested Places

Certain place forms are required to have one or more arguments which are themselves places. The prime example of this, and the only example from among built-in syntactic places, are DWIM forms. A DWIM form has the syntax

Note that not only is the entire form a place, denoting some element or element range of obj-place, but there is the added constraint that obj-place must also itself be a syntactic place.

This requirement is necessary, because it supports the behavior that when the element or element range is updated, then obj-place is also potentially updated.

After the assignment (set [obj 0..3] '("forty" "two")) not only is the range of places denoted by [obj 0..3] replaced by the list of strings ("forty" "two") but obj may also be overwritten with a new value.

This behavior is necessary because the DWIM brackets notation maintains the illusion of an encapsulated array-like container over several dissimilar types, including Lisp lists. But Lisp lists do not behave as fully encapsulated containers. Some mutations on Lisp lists return new objects, which then have to stored (or otherwise accepted) in place of the original objects in order to maintain the array-like container illusion.

8.7.9 Built-In Syntactic Places

The following is a summary of the built-in place forms, in addition to symbolic places denoting variables. New syntactic place forms can be defined by TXR programs.

8.7.10 Built-In Place-Mutating Operators

The following is a summary of the built-in place mutating macros. They are described in detail in their own sections.

8.8 Namespaces and Environments

TXR Lisp is a Lisp-2 dialect: it features separate namespaces for functions and variables.

8.8.1 Global Functions and Operator Macros

In TXR Lisp, global functions and operator macros coexist, meaning that the same symbol can be defined as both a macro and a function.

There is a global namespace for functions, into which functions can be introduced with the defun macro. The global function environment can be inspected and modified using the symbol-function accessor.

There is a global namespace for macros, into which macros are introduced with the defmacro macro. The global function environment can be inspected and modified using the symbol-macro accessor.

If a name x is defined as both a function and a macro, then an expression of the form (x ...) is expanded by the macro, whereas an expression of the form [x ...] refers to the function. Moreover, the macro can produce a call to the function. The expression (fun x) will retrieve the function object.

8.8.2 Global and Dynamic Variables

There is a global namespace for variables also. The operators defvar and defparm introduce bindings into this namespace. These operators have the side effect of marking a symbol as a special variable, of the symbol are treated as dynamic variables, subject to rebinding. The global variable namespace together with the special dynamic rebinding is called the dynamic environment. The dynamic environment can be inspected and modified using the symbol-value accessor.

The operators defvarl and defparml introduce bindings into the global namespace without marking symbols as special variables. Such bindings are called global lexical variables.

8.8.3 Global Symbol Macros

Symbol macros may be defined over the global variable namespace using defsymacro.

Note that whereas a symbol may simultaneously have both a function and macro binding in the global namespace, a symbol may not simultaneously have a variable and symbol macro binding.

8.8.4 Lexical Environments

In addition to global and dynamic namespaces, TXR Lisp provides lexically scoped binding for functions, variables, macros, and symbol macros. Lexical variable binding are introduced with let, let* or various binding macros derived from these. Lexical functions are bound with flet and labels. Lexical macros are established with macrolet and lexical symbol macros with symacrolet.

Macros receive an environment parameter with which they may expand forms in their correct environment, and perform some limited introspection over that environment in order to determine the nature of bindings, or the classification of forms in those environments. This introspection is provided by lexical-var-p, lexical-fun-p, and lexical-lisp1-binding.

Lexical operator macros and lexical functions can also coexist in the following way. A lexical function shadows a global or lexical macro completely. However, the reverse is not the case. A lexical macro shadows only those uses of a function which look like macro calls. This is succinctly demonstrated by the following form:

The (fun foo) and [fun] expressions are oblivious to the macro; the macro expansion process process the symbol foo in those contexts. However the form (foo) is subject to macro-expansion and replaced with 44.

8.8.5 Pattern Language and Lisp Scope Nesting

TXR Lisp expressions can be embedded in the TXR pattern language in various ways. Likewise, the pattern language can be invoked from TXR Lisp. This brings about the possibility that Lisp code attempts to access pattern variables bound in the pattern language. The TXR pattern language can also attempt to access TXR Lisp variables.

The rules are as follows, but they have undergone historic changes. See the COMPATIBILITY section, in particular notes under 138 and 121, and also 124.

A Lisp expression evaluated from the TXR pattern language executes in a null lexical environment. The current set of pattern variables captured up to that point by the pattern language are installed as dynamic variables. They shadow any Lisp global variables (whether those are defined by defvar or defvarl).

In the reverse direction, a variable reference from the TXR pattern language searches the pattern variable space first. If a variable doesn't exist there, then the lookup refers to the TXR Lisp global variable space. The pattern language doesn't see Lisp lexical variables.

When Lisp code is evaluated from the pattern language, the pattern variable bindings are not only installed as dynamic variables for the sake of their visibility from Lisp, but they are also specially stored in a dynamic environment frame. When TXR pattern code is reentered from Lisp, these bindings are picked up from the closest such environment frame, allowing the nested invocation of pattern code to continue with the bindings captured by outer pattern code.

Concisely, in any context in which a symbol has both a binding as a Lisp global variable as well as a pattern variable, that symbol refers to the pattern variable. Pattern variables are propagated through Lisp evaluation into nested invocations of the pattern language.

The pattern language can also reference Lisp variables using the @ prefix, which is a consequence of that prefix introducing an expression that is evaluated as Lisp, the name of a variable being such an expression.

9 LISP OPERATOR, FUNCTION AND MACRO REFERENCE

9.1 Conventions

The following sections list all of the special operators, macros and functions in TXR Lisp.

9.2 Form Evaluation

A compound expression with a symbol as its first element, if intended to be evaluated, denotes either an operator invocation or a function call. This depends on whether the symbol names an operator or a function.

When the form is an operator invocation, the interpretation of the meaning of that form is under the complete control of that operator.

If the compound form is a function call, the remaining forms, if any, denote argument expressions to the function. They are evaluated in left-to-right order to produce the argument values, which are passed to the function. An exception is thrown if there are not enough arguments, or too many. Programs can define named functions with the defun operator

Some operators are macros. There exist predefined macros in the library, and macro operators can also be user-defined using the macro-defining operator defmacro. Operators that are not macros are called special operators.

Macro operators work as functions which are given the source code of the form. They analyze the form, and translate it to another form which is substituted in their place. This happens during a code walking phase called the expansion phase, which is applied to each top-level expression prior to evaluation. All macros occurring in a form are expanded in the expansion phase, and subsequent evaluation takes place on a structure which is devoid of macros. All that remains are the executable forms of special operators, function calls, symbols denoting either variables or themselves, and atoms such as numeric and string literals.

Special operators can also perform code transformations during the expansion phase, but that is not considered macroexpansion, but rather an adjustment of the representation of the operator into an required executable form. In effect, it is post-macro compilation phase.

Note that Lisp forms occurring in TXR pattern language are not individual top-level forms. Rather, the entire TXR query is parsed at the same time, and the macros occurring in its Lisp forms are expanded at that time.

9.2.1 Operator quote

9.3 Variable Binding

Variables are associations between symbols and storage locations which hold values. These associations are called bindings.

Lexical environments hold local variables, and nest according to the syntactic structure of the program. Lexical bindings are always introduced by a some form known as a binding construct, and the corresponding environment is instantiated during the evaluation of that construct. There also exist bindings outside of any binding construct, in the so-called global environment. Bindings in the global environment can be temporarily shadowed by lexically-established binding in the dynamic environment. See the Special Variables section above.

Certain special symbols cannot be used as variable names, namely the symbols t and nil, and all of the keyword symbols (symbols in the keyword package), which are denoted by a leading colon. When any of these symbols is evaluated as a form, the resulting value is that symbol itself. It is said that these special symbols are self-evaluating or self-quoting, similarly to all other atom objects such as numbers or strings.

When a form consisting of a symbol, other than the above special symbols, is evaluated, it is treated as a variable, and yields the value of the variable's storage location. If the variable doesn't exist, an exception is thrown.

Note: symbol forms may also denote invocations of symbol macros. (See the operators defsymacro and symacrolet). All macros, including symbol macros, which occur inside a form are fully expanded prior to the evaluation of a form, therefore evaluation does not consider the possibility of a symbol being a symbol macro.

9.3.1 Operator defvar and macro defparm

9.3.2 Macros defvarl and defparml

9.3.3 Operators let and let*

9.4 Functions

9.4.1 Operator defun

9.4.2 Operator lambda

9.4.3 Macros flet and labels

9.4.4 Function call

9.4.5 Functions apply and iapply

9.4.6 Operator fun

9.4.7 Operator dwim

9.4.8 Function functionp

9.4.9 Function copy-fun

9.5 Sequencing, Selection and Iteration

9.5.1 Operators/functions progn and prog1

9.5.2 Macro/function prog2

9.5.3 Operator cond

9.5.4 Macros caseq, caseql and casequal

9.5.5 Macros caseq*, caseql* and casequal*

9.5.6 Macros ecaseq, ecaseql, ecasequal, ecaseq*, ecaseql* and ecasequal*

9.5.7 Operator/function if

9.5.8 Operator/function and

9.5.9 Macro/function nand

9.5.10 Operator/function or

9.5.11 Macro/function nor

9.5.12 Macros when and unless

9.5.13 Macros while and until

9.5.14 Macros while* and until*

9.5.15 Macro whilet

9.5.16 Macros iflet and whenlet

9.5.17 Macro condlet

9.5.18 Macro ifa

9.5.19 Macro conda

9.5.20 Macro whena

9.5.21 Macro dotimes

9.5.22 Operators each, each*, collect-each, collect-each*, append-each and append-each*

9.5.23 Operators for and for*

9.5.24 Macros doloop and doloop*

9.5.25 Macros sum-each, sum-each*, mul-each and mul-each*

9.5.26 Macros each-true, some-true, each-false and some-false

9.5.27 Macros each-prod, collect-each-prod and append-each-prod

9.5.28 Macros each-prod*, collect-each-prod* and append-each-prod*

9.5.29 Macros sum-each-prod, sum-each-prod*, mul-each-prod and mul-each-prod*

9.5.30 Operators block and block*

9.5.31 Operators return and return-from

9.5.32 Function return*

9.5.33 Macros tagbody and go

9.5.34 Macros prog and prog*

9.6 Evaluation

9.6.1 Function eval

9.6.2 Function constantp

9.6.3 Function make-env

9.6.4 Functions env-vbind and env-fbind

9.6.5 Functions env-vbindings, env-fbindings and env-next

9.7 Global Environment

9.7.1 Accessors symbol-function, symbol-macro and symbol-value

9.7.2 Functions boundp, fboundp and mboundp

9.7.3 Function makunbound

9.7.4 Functions fmakunbound and mmakunbound

9.7.5 Function func-get-form

9.7.6 Function func-get-name

9.7.7 Function func-get-env

9.7.8 Functions fun-fixparam-count and fun-optparam-count

9.7.9 Function fun-variadic

9.7.10 Function interp-fun-p

9.7.11 Function vm-fun-p

9.7.12 Function special-var-p

9.7.13 Function special-operator-p

9.8 Object Type

In TXR Lisp, objects obey the following type hierarchy. In this type hierarchy, the internal nodes denote abstract types: no object is an instance of an abstract type. Nodes in square brackets indicate an internal structure in the type graph, invisible to programs, and angle brackets indicate a plurality of types which are not listed by name:

That is to say, the types are exhaustively partitioned into atoms and conses; an object is either a cons or else it isn't, in which case it is the abstract type atom.

The cons type is odd in that it is both an abstract type, serving as a supertype for the type lcons and it is also a concrete type in that regular conses are of this type.

The type nil is an abstract type which is empty. That is to say, no object is of type nil. This type is considered the abstract subtype of every other type, including itself.

The type nil is not to be confused with the type null which is the type of the nil symbol.

Because the type of nil is the type null and nil is also a symbol, the null type is a subtype of sym.

9.8.1 Function typeof

9.8.2 Function subtypep

9.8.3 Function typep

9.8.4 Macro typecase

9.8.5 Macro etypecase

9.8.6 Function built-in-type-p

9.9 Object Equivalence

9.9.1 Functions identity, identity* and use

9.9.2 Functions null, not and false

9.9.3 Functions true and have

9.9.4 Functions eq, eql and equal

9.9.5 Functions neq, neql and nequal

9.9.6 Functions meq, meql and mequal

9.9.7 Function less

9.9.8 Function greater

9.9.9 Functions lequal and gequal

9.9.10 Function copy

9.10 List Manipulation

9.10.1 Function cons

9.10.2 Function atom

9.10.3 Function consp

9.10.4 Accessors car and first

9.10.5 Accessors cdr and rest

9.10.6 Functions rplaca and rplacd

9.10.7 Accessors second, third, fourth, fifth, sixth, seventh, eighth, ninth and tenth

9.10.8 Functions append and nconc

9.10.9 Function append*

9.10.10 Functions revappend and nreconc

9.10.11 Function list

9.10.12 Function list*

9.10.13 Accessor sub-list

9.10.14 Function replace-list

9.10.15 Functions listp and proper-list-p

9.10.16 Function endp

9.10.17 Function length-list

9.10.18 Function copy-list

9.10.19 Function copy-cons

9.10.20 Function copy-tree

9.10.21 Functions reverse and nreverse

9.10.22 Accessor nthlast

9.10.23 Accessor butlastn

9.10.24 Accessor nth

9.10.25 Accessor nthcdr

9.10.26 Function tailp

9.10.27 Accessors caar, cadr, cdar, cddr, ..., cdddddr

9.10.28 Functions cyr and cxr

9.10.29 Functions flatten and flatten*

9.10.30 Functions flatcar and flatcar*

9.10.31 Function tree-find

9.10.32 Functions memq, memql and memqual

9.10.33 Functions member and member-if

9.10.34 Functions rmemq, rmemql, rmemqual, rmember and rmember-if

9.10.35 Functions conses and conses*

9.10.36 Function delcons

9.11 Association Lists

Association lists are ordinary lists formed according to a special convention. Firstly, any empty list is a valid association list. A nonempty association list contains only cons cells as the key elements. These cons cells are understood to represent key/value associations, hence the name "association list".

9.11.1 Function assoc

9.11.2 Functions assq and assql

9.11.3 Functions rassq, rassql and rassoc

9.11.4 Function acons

9.11.5 Function acons-new

9.11.6 Function aconsql-new

9.11.7 Function alist-remove

9.11.8 Function alist-nremove

9.11.9 Function copy-alist

9.11.10 Function pairlis

9.12 Property Lists

A property list, also referred to as a plist, is a flat list of even length consisting of interleaved pairs of property names (usually symbols) and their values (arbitrary objects). An example property list is (:a 1 :b "two") which contains two properties, :a having value 1, and :b having value "two".

An improper plist represents Boolean properties in a condensed way, as property indicators which are not followed by a value. Such properties only indicate their presence or absence, which is useful for encoding a Boolean value. If it is absent, then the property is false. Correctly using an improper plist requires that the exact set of Boolean keys is established by convention.

In this document, the unqualified terms property list and plist refer strictly to an ordinary plist, not to an improper plist.

9.12.1 Function prop

9.12.2 Function memp

9.12.3 Functions plist-to-alist and improper-plist-to-alist

9.13 List Sorting

Note: these functions operate on lists. The principal sorting function in TXR Lisp is sort, described under Sequence Manipulation.

The merge function described here provides access to an elementary step of the algorithm used internally by sort when operating on lists.

The multi-sort operation sorts multiple lists in parallel. It is implemented using sort.

Operators	Class	Associativity
`(R) []`	primary
`R? R+ R* R%...`	postfix	left-to-right
`R1R2`	catenation	left-to-right
`~R ...%R`	unary	right-to-left
`R1&R2`	intersection	left-to-right
`R1\|R2`	union	left-to-right

Manpage for TXR

May 10, 2022

Contents

1 NAME

2 SYNOPSIS

3 DESCRIPTION

4 ARGUMENTS AND OPTIONS

5 STATUS AND ERROR REPORTING

6 BASIC TXR SYNTAX

6.1 Comments

6.2 Hash-Bang Support

6.2.1 Basic Hash Bang

6.2.2 Argument Generation with --args and --eargs

6.2.3 Argument Generation with the Null Hack

6.2.4 Passing Options to TXR via Hash-Bang Null Hack

6.2.5 Hash Bang and Setuid

6.3 Whitespace

6.4 Text

6.5 Special Characters in Text

6.6 Character Handling and International Characters

6.7 Regular Expression Directives

6.8 Variables

6.9 Negative Match

6.9.1 Variable Followed by Nothing

6.9.2 Variable Followed by Text

6.9.3 Variable Followed by a Function Call or Directive

6.9.4 Consecutive Variables

6.9.5 Consecutive Variables via Directive

6.9.6 Longest Match

6.10 Positive Match

6.11 Special Symbols nil and t

6.12 Keyword Symbols

6.13 Regular Expressions

6.14 Compound Expressions

6.15 Character Literals

6.16 String Literals

6.17 Word List Literals

6.18 String Quasiliterals

6.19 Quasiword List Literals

6.20 Numbers

6.21 Comments

7 DIRECTIVES

7.1 Overview

7.2 Subexpression Evaluation

7.3 Input Scanning and Data Manipulation

7.3.1 The next directive

7.3.2 The skip directive

7.3.3 Reducing Backtracking with Blocks

7.3.4 The trailer directive

7.3.5 The freeform directive

7.3.6 The fuzz directive

7.3.7 The line and chr directives

7.3.8 The name directive

7.3.9 The data directive

7.3.10 The eof directive

7.3.11 The some, all, none, maybe, cases and choose directives

7.3.12 The require directive

7.3.13 The if directive

7.3.14 The Lisp if versus TXR if

7.3.15 The gather directive

7.3.16 The until / last clause in gather

7.3.17 Keyword parameters in gather

7.3.18 The collect directive

7.3.19 The until / last clause in collect

7.3.20 Keyword parameters in collect

7.3.21 Specifying variables in collect

7.3.22 Mandatory until and last

7.3.23 The coll directive

7.3.24 Keyword parameters in coll

7.3.25 The flatten directive

7.3.26 The merge directive

7.3.27 The cat directive

7.3.28 The bind directive

7.3.29 Keywords in the bind directive

7.3.30 Lisp forms in the bind directive

7.3.31 The set directive

7.3.32 The rebind directive

7.3.33 The forget directive

7.3.34 The do directive

7.3.35 The mdo directive

6.2.2 Argument Generation with `--args` and `--eargs`

6.11 Special Symbols `nil` and `t`

7.3.1 The `next` directive

7.3.2 The `skip` directive

7.3.4 The `trailer` directive

7.3.5 The `freeform` directive

7.3.6 The `fuzz` directive

7.3.7 The `line` and `chr` directives

7.3.8 The `name` directive

7.3.9 The `data` directive

7.3.10 The `eof` directive

7.3.11 The `some`, `all`, `none`, `maybe`, `cases` and `choose` directives

7.3.12 The `require` directive

7.3.13 The `if` directive

7.3.14 The Lisp `if` versus TXR `if`

7.3.15 The `gather` directive

7.3.16 The `until` / `last` clause in `gather`

7.3.17 Keyword parameters in `gather`

7.3.18 The `collect` directive

7.3.19 The `until` / `last` clause in `collect`

7.3.20 Keyword parameters in `collect`

7.3.21 Specifying variables in `collect`

7.3.22 Mandatory `until` and `last`

7.3.23 The `coll` directive

7.3.24 Keyword parameters in `coll`

7.3.25 The `flatten` directive

7.3.26 The `merge` directive

7.3.27 The `cat` directive

7.3.28 The `bind` directive

7.3.29 Keywords in the `bind` directive

7.3.30 Lisp forms in the `bind` directive

7.3.31 The `set` directive

7.3.32 The `rebind` directive

7.3.33 The `forget` directive

7.3.34 The `do` directive

7.3.35 The `mdo` directive

7.3.36 The `in-package` directive

7.4.2 The `block` directive

7.4.6 Interaction Between the `trailer` and `accept` Directives

7.4.7 Interaction Between the `next` and `accept` Directives

7.4.8 Interaction Between Functions and the `accept` directive

7.4.9 Interaction Between `finally` and the `accept` directive

7.4.10 Vertical-Horizontal Mismatch Between `block` and `accept`

7.4.11 Horizontal-Horizontal Mismatch between `block` and `accept`

7.6.1 The `load` and `include` directives

7.7.2 The `output` directive

7.7.7 The `repeat` directive

7.7.8 Nested `repeat` directives

7.7.9 The `rep` directive

7.7.10 `repeat` and `rep` Examples

7.7.11 The `close` directive

7.7.14 The `deffilter` directive

7.7.15 The `filter` directive

7.8.2 The `try` directive

7.8.3 The `finally` clause

7.8.4 `catch` clauses

7.8.5 `catch` Clauses with Parameters

7.8.6 The `throw` directive

7.8.7 The `defex` directive

7.8.8 The `assert` directive