TXR 294 2024-03-17 Features - Compiler: - Now generates inline code for chained functions and lambdas, such as that arising out of the opip syntax. - OOP: - new special method length-< lets a lazy sequence object implement support for being tested with the length-< function (is the length less than a given integer without forcing existence). - Lib: - New hist-sort-by function: hist-sort with a mapping function. - New functions cons-count and cons-find. - New function rangeref for obtaining an intermediate value from a range using an integer index as if it were a sequence. - For consistency: because ranges can be treated as sequences. - Pattern matching: - End pattern in @(sme) and @(end) can be any pattern now, not a list pattern. - Internals: - New framework introduced for gathering items into a sequence, called "seq_iter". - Takes into account the type of sequence being constructed up-front, to avoid the overhead of collecting a list and then coercing it. - Functions that have been improved with seq_build and in some cases seq_iter: keep-keys-if, separate, separate-keys, tuples, mapcar, mappend, mapdo, partition-by, partition-if, partition, split, split*, zip, unique, find, rfind, window-map, append, nconc. Bugs - stability: - gc problem in iter-begin. - gc problem in sub-str on lazy string arg. - gc issue in eval in code related to binding special variables. - gc correctness issues in make-strlist-input-stream, make-struct, make-lazy-struct, copy-search-tree, make-similar-tree, copy-fun (for VM functions). - build: - fix in configure script for BSD grep. - fixes in ordering issues in compilation of stdlib to actually achieve the shortest bootstrapping time. - ports: - fixes for OpenBSD. - Support PPC64 with Altivec more properly by saving and restoring VR31 register. Fixes a reported failure to build and pass tests. - Not a perfect fix: pragmatic whack-a-mole approach based on empirically seeing which Altivec registers occur in the code, needing saving. - cygwin: - fixed broken handling of drive-relative paths. - handling of termination status in run and sh functions fixed to be like that on POSIX platforms. - documentation: - source parameter of mmap documented. - mmap: too low length confusingly diagnosed: as "zero-sized element type". - hash-eql: always returning zero due to regression. - mapcar, mappend, mapdo: avoid alloca request proportional to number of args, switching to malloc over a certain size. - list-vec: accidental use of int type to hold vector length: fail on 64 bit systems with vectors having more than 4G entries. - misc: - avoid realloc to size zero, since latest ISO C draft makes it undefined behavior. - keep-if: don't report as remove-if in errors. TXR 293 2023-12-28 Features - load: - when a top-level load exits by returning from the load block, the block value now becomes termination status of process. - JSON: - now supports Lisp comments as an extension. - I/O: - New functions read-objects, file-get-objects, file-put-objects, file-append-objects. - Hash Tables: - New function hash-join, similar to hash-uni but usefully different. Bugs - TXR: bug in @{nil ...} variable match. - compiler: - compile-file can now handle top-level forms that perpetrate a non-local exit, without requiring compile-only around them. - optimizer now watches for exceptions during constant folding, similarly to how the front end does. - listener: - several bugs in auto compound expr mode fixed. - perm, rperm, comb, rcomb functions: - now support all sequence types, not only lists, vectors and strings. - comb was skipping combinations: defective function repaired. - glob: - suppress duplicates generated when multiple double star patterns are present (supported by the recent glob*). - fix memory leak that can happen on early exit due to error function performing a nonlocal control transfer. - sh-esc family of functions (recently introduced): - incorrect specifications overhauled. - (rcons ...) print/read consistency issue fixed. - Cygwin: termination status returned by (sh ...) and (run ...) now has the correct integer value, or nil for abnormal exit, like on Linux. TXR 292 2023-11-20 Features - Build: - speed up compilation of stdlib with optimal compilation order of several key files. - Revised confusing build instructions in ./configure script. - Makefile improvements: - cruft removed; dependency generation simplified. - mkdir invocations factored out to rule. - DELETE_ON_ERROR: used now. - Streams: - A new protocol between the close-stream function and its lower-level implementation allows delegate streams to implement a reference counting discipline for closing. - open-file now supports a "T" mode for O_TMPFILE (on Linux, or anywhere else O_TMPFILE is supported). - Sequences: - New functions nested-vec and nested-vec-of for easily constructing (simulated) multi-dimensional arrays. - New mref accessor for multi-dimensional access. - also has a side job calling curried functions. - New csort-group function: - Like sort-group, but uses csort instead of sort for key caching. - New hist-sort function: make histogram of sequence, and sort by descending frequency. - New length-list-< function for testing whether a list is shorter than a given length without traversing it beyond that length. - New length-< function for testing any sequence's length against a value without traversing/forcing the sequence to calculate its length. - ref now documented as accessor, not just function. - del operator now allowed on places that index into sequences implemented by structures: - requires both lambda and lambda-set methods. - Strings: - New str-esc function for generic character escaping. - New shell escaping functions: sh-esc, sh-esc-all, sh-esc-dq and sh-esc-sq. - Packages - New feature: local symbol renaming - Symbols can be imported under alternative names. - New use-sym-as function and :use-syms-as defpackage clause. - Better idea than nicknaming packages (package-local nicknames). - Functional library: - New: left-inserting pipeline operators: lflow, lopip, loand. - New: macros orf and lorf for condensing certain op syntax. - New: tap macro, for expressing side effects in pipelines while passing through the value. - Syntactic places: - New: ensure macro evaluates an expression and stores its value into a place, if that place's current value is nil. - Pattern matching: - some match-case instances are now transformed into casequal. - FFI: - Now provides a setjmp macro, longjmp function as well as a jmp-buf utility function that allocates a jmp-buf. - Now possible to interact with libraries that use longjmp for error aborts like libpng and such. - POSIX: - chdir function takes stream or file descriptor, which are handled via fchdir. - New glob* function to complement glob: - glob* supports the ** (double star) operator for recursing, via its own implementation, not relying on glob providing that. - glob* has its own implementation of brace expansion. - New rlink function which is like link, but resolves the target path if it denotes a symlink. - Listener: - *listener-sel-inclusive-p* is now default, since most terminals have block cursors by default. - New feature: auto compound expression mode lets you omit the parentheses. - Math: - The tofloat and toint functions are now generic via the user-defined arithmetic struct mechanism. - JSON: - Allow integer objects to be printed rather than insisting on numbers being floating-point. - Allow lists (including lazy lists) to print as [ ... ] rather than insisting on vectors. - Windows Installer: - Does not require admin privileges any more; will install for just the current user if the user isn't admin. - Documentation: - big change: new hashing scheme for navigation and doc lookup. - section titles now hashed in a more robust way that is resistant against most kinds of edits. - stdlib/doc-syms.tl file is now gone: one less thing to maintain. - Numerous fixes in manual. Bugs - Compiler: - numbers now externalized sanely in .tlo files: - compile-file makes sure base 10 is used for integers - floating point numbers written with sufficient precision - Loading: - load-args-process bugfix: :compile action must load. - Streams: - close-stream caches only successful result from underlying function. - a few close functions underneath close-stream were returning nil in the successful case. - OOP: - Fix segfault looking up special method after the static slots table of the type has been resized. - Pattern matching: - Missing autoloads for match-error and sys:match-pat-error, causing compiled files containing pattern matching not to load. - Awk macro: - prn returns nil - Search trees: - bug: tree-delete-specific-node not using key fun. - Lib: - Two bugs in flatten* function. - Bug in deletion of (ref ...) place: incorrect when object is list. - Crypt: - less strict error token detection for wider platform support. - Time: - time-utc and time-local methods must subtract gmtime, not add. - time-parse-utc and time-parse-local, likewise. - Build: - Misspelled PLATFORM_LDLIBS corrected. - Fix for _TIME_BITS being tied to _FILE_OFFSET_BITS - Address autoload circular dependency involving stdlib/error.tl that can fail builds under some library build orders. - Vim: - Fix lack of recognition for char escapes in quasilit. TXR 291 2023-08-06 Features - opip macro: - now supports binding variables which are visible to the rest of the chain. Bugs - symbol-function accessor: not supporting functions whose names are not symbols, like (meth ...) - Regression introduced in TXR 290. - compiler: invalid constant folding of load-time values. - Recent regression. - Pattern Language: in output-side @(repeat), inability to specify :vars together with :counter. - Regression dating back to March 2016. TXR 290 2023-07-29 Features - Compiler: - Deeper constant-folding optimizations, assisted by data flow information. - Data Integrity: - SHA-1 digest now provided; it is in wide use. - Lib: - group-by, group-reduce and unique functions refactored to use sequence iteration. - Math: - Numerous C99 math functions now exposed: - cbrt, erf, erfc, exp10, exp2, expm1, gamma, j0, j1, lgamma, log1p, logb, nearbyint, rint, significand, tgamma, y0, y1, copysign, drem, fdim, fmax, fmin, hypot, jn, ldexp, nextafter, remainder, scalb, scalbln, and yn - All of these can be user-defined in a custom arithmetic struct. - Environments - Internal representation of top-level environments is simplified, eliminating a level of indirection. - New functions for lexical introspection: - lexical-binding-kind, lexical-fun-binding-kind, lexical-symacro-p and lexical-macro-p. Bugs - hash: out-of-bounds access in hash-iter-peek function causing instability in hash iteration that depend on this function. - crypt function: - handle libxcrypt failure tokens properly, so that we treat them as an error instead of returnig them to the application. - md5: was totally broken on big endian. - VM: compiled functions were not picking up redefinitions of functions that they call, sticking with the old functions. This issue was uncovered during work on the new representation of top-level environments, which also happens to fix the problem. - pattern matching: - the above VM bug fix means that the double definition of sys:non-triv-pat-p in the pattern matching module is handled right; causing sys:transform-qquote to be expanded using the correct definition of the function, exposing an obscure issue with quasiquoted hash literals. - another bug: variable patterns like @var were not seeing lexical symbol macros, only global symbol macros, global variables and lexical variables. - environments: redefinition of symbol macros was found not to trigger cache invalidation in VM descriptors, causing compiled code to keep seeing the old definitions of the symbol. - lib: the del operator now works with index lists, which is the access notation like [obj '(1 2 4)] for picking out indices. As a result of this, there is a change in semantics in how the replace function handles index lists. If there are fewer replacement items than indices, then instead of stopping, the replace function deletes the specified indices. (Subject to -C option.) - paths: rel-path function was considering empty strings to be absolute paths, causing (rel-path "" "a") to be diagnosed as an invalid mixture of absolute and relative paths. - environments: bugfix in lexical-var-p and lexical-lisp1-binding. TXR 289 2023-07-02 Features - TXR Pattern Language: - New @(push) directive uses @(output) syntax to push lines back into input. - Pattern Matching: - New match-cond macro. - Lib: - New cached sorting functions, for situations when a fairly expensive keyfun is used for sorting. - Lisp: - eval takes macro environment. - integers and ranges are function-callable objects - provides succinct indexed access in functional expressions, using integers and ranges as higher-order functions. - Hash Tables: - Switched from chained hashing to open-addressing with linear probing. - New hash-map function: populate a hash with keys from a sequence, and values from a function over those keys. Bugs - hash tables: bug in initial hash mask calculation, caused zero bits, causing some chains not to be used, reducing efficiency. - gc bug in vector case of ssort function, causing crash. - equal: fix broken cases in equality substitution. - range objects were not treated as iterable in some situations, for no good reason: e.g. (take 13 "AA".."BB") didn't work, requiring (take 13 (list-seq "AA".."BB")) as a workaround. TXR 288 2023-06-10 Features - Lib: - New keep-keys-if, separate-keys functions. - Compiler: - New clean-file function for removing compiled file with name built-in path resolution strategy harmonizing with compile-file. - New compiler option log-level. - level 1: info message when file is compiled. - level 2: info message when top-level form is compiled. - Modularization: - New functions load-args-recurse and load-args-process - Handle a build/load command passed in *load-args*. - Provide a disciplined way to structure programs into hierarchical library modules. Bugs - Obscure in bug in compile-file giving rise to a .tlo that cannot be loaded, under certain conditions. - Autoload issue affecting with-compilation-unit, causing compiled files that use with-compilation-unit not to load. TXR 287 2023-06-03 Features: - Lisp: - New: progv special operator, similar to Common Lisp's - New: compiler-let: binds dynamic variables in the compiler's context, allowing control over the compiler at the expression level. - with-compile-options now implemented using compiler-let. - Awk macro: - redirection operators visible in wider scope - new :fun clause for binding functions local to macro. - Compiler: - small optimizations: when all local functions in a labels/flet block are unused, the frame is not generated for them. - Expander: - Parameter list macros now work in nested lambda lists. (This is also listed below under Bugs.) - New expander-let macro for binding special variables at macro-expansion time. - Allows customization of macros which occur inside, by having them respond to the values of the specials. - Used in TL-WHO port of CL-WHO to fix CL-WHO bugs. - Command line: - The -e option evaluates multiple expressions from the same argument string. - They are read together before evaluation, almost as if they were in the same progn. - Listener: - Evaluates multiple expressions in command line, instead of complaining about trailing material. - Lib: - load function has new features: - extra arguments may be passed to load, which are dynamically bound to a special var called *load-args* - a loaded file can bail early using using (return-from load) or (return-from load ) - the interrupted load function will then return that value to its caller. - thus loaded files can behave like functions with arguments and return values. Bugs - Android: fixes for running on Android 13 via Termux. - Environments: - Fixed crash when certain built in variables are removed with makunbound. - Fixed (symbol-value ...) wrongly storing a value to the top-level binding rather than the current dynamic binding. - Fixed bug in the VM: getlx and setlx instructions using dynamic lookup rather than global. - Expander: - Fixed bug in empty case of flets/labels causing unrelated symbol macros to be strangely affected. - Parameter list macros now work in nested lambda lists. - Awk macro: fixed completely broken redirection operators. - Parser: - There is now a proper handler for fatal Flex errors, like when a token is ridiculously long. - Flex-generated default handler prints something and exits. - Our handler throws exception. TXR 286 2023-05-07 Features - Hash tables: - some internal code improvements/streamlining - new hash-props function from instantiating a table from alternating key/value arguments, requiring no temporary list to be consed up. - Sorting: - New ssort and snsort functions: these are counterparts of sort and nsort which are stable on vector-like sequences. - The nsort and sort function's quicksort implementation now uses the Hoare partitioning scheme instead of Lomuto: - Observed a 21% improvement sorting a randomized vector of a million items. - The quadratic behavior on a sequence consisting of a repeated item is gone. - Time: - New time-str-local and time-str-utc functions, which reverse the arguments, for better partial application. - the time argument in time-{fields,struct}-{local,utc} is now optional; if omitted, the current time is used as if by calling (time). - Structs: - Small improvement in defstruct: if boa arguments are defined referencing slots that don't exist in any struct, this is now diagnosed. - Compiler: - Lots of new optimization work. There is now one more optimization round, and *opt-level* now goes up to 6 rather than 7, 7 being the new default value. This release makes 23 compiler commits. Bugs - Fixed incorrect scope in conda/condlet. - Fixes for regressions preventing the source code of of stdlib/ being used (.tl files, not .tlo), which is needed for debugging some TXR problems. - Fixed issue that happens when code is loaded that generates warnings during error exception processing, causing an "invalid re-entry of exception logic", interfering with debugging TXR using an uncompiled library (.tl files rather than .tlo). - Fixed another issue using .tl files: interference between library loading and the -C compat option. Compatibility is temporarily disabled while auto-loading. - build macro: code rearranged to eliminate circular dependency, preventing modules which depend on the macro from loading. - Pattern matching: ^#S() and ^H(()) quasiquote patterns work now, thanks to a change in the parser. - Compiler: - Fixed incorrect evaluation order of function arguments (when local variables are involved that are subject to side effects during evaluation). - Fixed issue with compiling defmacro: - entire macro form was being retained - yet errors not reported against the correct operator name: e.g. (defstruct) says that defmacro has insufficient arguments. - fix also affects tree-bind and other operators. - Fixes September 2022 regression in liveness calculation, causing certain optimizations to be forgone. - Fixed incorrect blind register renaming across the arguments of a close instruction, which are not actually source operands of that instruction. - Latent problem exposed when trying to replace V register by T register more aggressively. - Fixed incorrect live register calculation across catch instruction. - This has two clobber register operands. - Representation of instruction live info was expanded to handle two register defs. TXR 285 2023-03-28 Features - Lib: - time (on platforms that have a timezone field in struct tm): - functions which convert a destructured time into a numeric time, like time-parse-utc, now take the time zone into account, and add a displacement. - functions which format time now via strftime now set the time zone field in the underlying struct tm, so that the %z specifier featured in glibc's strftime can be meaningfully used. - New function: ignore: synonym of nilf, intended for suppressing unbound variable warnings - New function: arithp: tests for arithmetic objects, including ranges and structures with + method. - range/range*: these functions now support non-arithmetic types: e.g (range "AAA" "ZZZ" 2) generates ("AAA" "AAC" ... "ZZW" "ZZY"). - TXR Pattern Language: - fix exception being thrown in matching a bound variable whose value is a lazy list of strings rather than ordinary list. - e.g. value captured with @(data ...) - Structural Pattern Matching: - @nil is now supported in predicates. - @(< @nil 42) is like @(< @a 42) but no variable is bound. - Syntax: - The symbol t can be used in macro parameter lists and tree-bind, to specify a pseud-variable which just throws away the corresponding value. - Compiler: - New options mechanism: - compile-options struct type - *compile-options* special variable. - with-compile-options macro - New unused variable warnings are on by default. Bugs - build: fixed regression in building without CONFIG_GEN_GC or CONFIG_DEBUG_SUPPORT, which are 1 by default. - gc: - premature reclamation bug in lisp_parse_impl which is used internally and as the implementation for functions that parse Lisp, regex and JSON. - premature reclamation bug in implementation of FFI enum types: neglect to traverse a struct member during gc marking. - premature bug in constructor for FFI structs: neglect to protect member types from gc in the loop that processes struct members. - printer: - [] object now prints as [] rather than [. nil], which isn't incorrect, just ugly. - search tree objects now print as #T(...) beyond the maximum printing depth, just like #H(...) and others. - Vim: - Fixed syntax highlighting for decimal integers and uninterned symbols. TXR 284 2022-12-30 Features - OOP: - new :inherit clause in defstruct so that inheritance bases can be specified by clauses. - Motivated by clauses being programmable. - Allows defstruct clause macro to bring in bases. - new feature: struct preludes. - preludes can specify clauses to inject into specific defstruct definitions (that have not yet been processed), without those definitions mentioning anything. - purely a macro-expansion-time feature. - Lib: - cat-str/join/join-with now allow nested sequences. - System Functions: - ftw function: the flags argument now defaults to ftw-phys if omitted (do not follow symbolic links). - compiler: - optimizations around catch - Awk macro: - result of condition in condition-action clause is avaialble via a new Awk variable named res. Bugs - Compilation from command line via --compile now sets the self-path variable. - Listener: drop security checks on Windows, where they don't work and generate false positives. - They are geared toward a multi-user system with a bona fide POSIX security and file permission model. - crypt: remove dubious validator. - cannot reproduce the crash issue it was supposed to work around. - read-once: now supports global variables properly. - crypt: fix ridiculous stack usage, caused by giant context structure for glibc's crypt_r. - hashing: negative floating-point zero handled. - math: expt with a zero exponent yields 1.0. - though works that way already on all platforms, it is now documented and assured. - compiler: - some functions were constant folded that must not be, because they are required to allocate fresh objects each time they are called. - an instance of runaway recursion in the compiler was fixed in constant-folding code. TXR 283 2022-10-16 Features - Low Level: - NaN boxing now works on Android, in spite of its pointer tagging. - String objects no longer track their storage allocation size on platforms that have malloc_usable_size. - The word of storage in a string object thus made available has not yet been put to a use. - Lisp: - New %fun% symbol macro provides name of current function. - Separator commas are now allowed in numeric tokens. - New functions - macroexpand-params: - expand parameter list macros made with define-param-expander. - macroexpand-place: - expand place macros made with define-place-macro - macroexpand-match: - expand macro patterns made with defmatch. - macroexpand-struct-clause: - expand defstruct macro clause made with define-struct-clause. - Small performance improvements in function dispatch. - functions with optional arguments no longer put through slow path - this could be listed under Bugs below - helper functions for fixed argument dispatch cases now handle more cases themselves rather than defer to slow path. - use of alloca has been eliminated from the creation of arguments on the stack in cases when the size is statically know. - OOP: - new :postfini clause in defstruct, allowing for finalization with order opposite to :fini - relaxation of constraint: defstruct can specify multiple :init, :fini, :postinit and :postfini clauses. - optional arguments :delegate clause now have init expressions that are not ignored, but specify the default value. - thus delegates can now customize the defaulting of optionals rather than being stuck with the target's behavior. - when delegates specify an optional parameter that corresponds to a non-optional target parameter, they can thus now specify a default value, rather than being stuck with nil. - Networking: - New sockaddr-str function: parse various textual address types into appropriate type of sockaddr. - New str-addr method in every sockaddr structure, for generating textual address. - I/O streams: - new inc-indent-abs function for incrementing absolute indentation, not relatively to current horizontal position. - JSON: - JSON printing now uses "standard-style" formatting, if a the newly introduced *print-json-format* variable is set/bound to the value :standard. Bugs - build: ./reconfigure issue when ./configure is interrupted. - str-in6addr bug. - hash: don't trim hash seed to 32 bits on 64 bit platforms. - JSON: restore stream indentation state if exception occurs during JSON printing. TXR 282 2022-09-16 Features: - New [. expr] syntax. This is also a bugfix because we have been printing (dwim . @sym) as [. @sym] without being able to read that syntax (read-print consitency issue). - NaN boxing representation for Lisp values. - enabled by ./configure --nan-boxing - 64 bit platforms only. - allows floating-point values not to be heap-allocated, Bugs: - compiler: - incorrect scopeing for init expressions of optional parameters. - bug in dead-code elimination causing compile-time exception. - bug in optimizer affecting code generated by prof operator, leading to a wrong result value. - compiler now diagnoses if there are too many variables added to a lexical frame (more than 1024). - numeric ranges in sequence iteration (seq-begin) now work with floating-point values. TXR 281 2022-09-03 Features: - Lib: - New search-all library function: like search but finds all matches. - New macro: close-lazy-streams: creates a dynamic contour of code which closes all streams that were bound to lazy conses during its execution. - TXR Pattern Language: - The @(next) directive now supports a :noclose modifier. - Because @(next) now closes the stream when it's done processing (see Bugs below), this new feature is required to opt-out of that behavior. - Vim: - Improvements to syntax highlighting definitions. Bugs - ffi: now defends against out-of-range wchar_t values being converted to Lisp character type. - TXR Pattern Language: - When a subquery opens a stream as a data source, that is now closed when that subquery is finished processing. TXR 280 2022-08-09 Bugs - Listener: - Fix regression: ~/.txr_history not loading unless ~/.txr_profile exists. - Build: - Handle failing hard link operation in "make install" so things work on Android again. TXR 279 2022-08-08 Features - Lib - missing count function added. - regsub - now accepts a string in place of the regex, - avoids consing a list of pieces to be catenated; works using string-extend. - gcd function rewritten for efficiency - when arguments fit into a machine word, bignum math is avoided - Build/Deployment - make install now creates hard links to the txr executable called txrlisp and txrvm, useful in scripting with unsuffixed files. - txrlisp behaves much like txr --lisp - txrvm behaves much like txr --compiled - compile-file translates txrlisp to txrvm in hash bang line. - Path test functions: - All path test functions now use effective UID not real. - New function path-components-safe for validating permissions along an entire path. - Useful for testing whether a path that is supposed to be private is actually properly secured. - Listener: - Security checks on .txr_history and .txr_profile have been revised. - Now done with help of path-components-safe in additition to path-private-to-me-p. Bugs - compile-file: - tries unsuffixed path before adding .tl, like load. - only tries different names on nonexistence error. - other exceptios now propagate out of the function. TXR 278 2022-07-01 Features - New str function for making a string filled with repeating pattern. - Syntax: stricter check in for/for* loop syntax. - I/O: - open-fileno (TXR's "fdopen") now takes pid argument, to associate the resulting stream with a process. - close-stream will subsequently wait on that process, and convert the status to a return value or exception. - all I/O convenience functions like command-get-linews now have a mode-opt argument. - For instance "z" can be used for compressed I/O. Bugs - Build: broken when no HAVE_ZLIB. - Command line: broken --free-all option. - String output streams: GC issue, occuring in some builds. - Listener: properly handle warnings coming out of code that is autoloaded during Tab completion. - Issue seen when working on TXR, with library .tlo files removed, so .tl files are used. - Compiler: failure in optimizer. - Compression: missing "z" support in open-command. - Missing: mode-opt argument of file-get-lines now implemented. TXR LZ77 :-) 2022-05-31 Features - Zlib integration: - New "z" mode option in open-file and open-fileno for Deflate compression (reading and writing): reads and writes gzip-compatible files. - Supported in convenience functions like file-put-string, file-get-buf and all those. - buf-compress and buf-decompress functions. - .tlo.gz files recognized by load as compressed. - .tlo.gz files may be catenated just like .tlo files. - Lib: - tok-str function takes count argument. - new spln and tokn functions: like spl and tok, but take a count argument limiting pieces returned. Bugs - tests: load-search test when run as superuser. - configure: don't exit when mmap isn't detected. - stream-set-prop: return t when :name prop set on file stream. - compilation bug in gc.c if HAVE_VALGRIND is on (maintainer mode only). - removed workaround for old Cygwin bug in I/O streams. - tags.tl renamed to txr-tags.tl to avoid name clash with tags file. - cygwin: sh function was wrongly using cmd.exe /c. TXR 276 2022-05-24 Features - Syntax: - printer: now nicely prints (a . @b) rather than (a sys:var b) and (a . @(e)) rather than (a sys:expr (e)). - Macros: - Subtle new expansion rule allows for more thorough expansion in situations when a macro and function are defined for the same symbol: - When a macro expands into the same-named function call, and that function call's arguments undergo expansion, the result is tried again as the original macro. - Command line: - New command line option --compile allows compile-update-file to be invoked more directly, without having to encode a Lisp expression as a command argument. - New command line option --in-package allows a package switch to take place within the command line. This is used by the compilation of stdlib, which takes place in the sys package. - In relation to command line: the message during the handling of an error exception encouraging the --backtrace option to be used is removed. This was a nuisance, and appeared in deployed programs that don't offer such an option. - Lib: - split-str now has a count parameter, to limit how many pieces are produced. When the split doesn't use the entire string, the remainder appears as a piece. This improvement was suggested by Paul. A. Patience in January 2022. - The spl function as well as tok-str could use this too; that is postponed to another release. - New trim-path-seps function for removing trailing path separators from a path. - FFI: - 64 bit bitfields are now supported. this means integer types which are 64 bits wide can now be be used as the basis of a bitfield, which can therefore be specified as 0 to 64 bits wide. - align operator now only increases alignment. - New pack operator for packing. - (pack (struct s ...)) syntax allows for all members of a struct to be packed. - The endian types like be-uint32 or le-uint16 can now be used as bit fields. - The layout takes place like on the machine of that endian; e.g. be-uint32 bitfields are filled most-significant-bit first. - If a bitfield follows a member of opposite endian, it starts a new storage cell in a fresh byte. Bugs - lambda-match: issue with variadic pattern. - FFI: bug: all unions were marked as incomplete types. - FFI: bug: empty structs/unions had an alignment of 0; should be 1. - FFI: bug: null terminated strings didn't work as flexible arrays. - copy-path-rec: didn't like trailing slash on source path. - FFI: alignment bug: arrays without a dimension were all treated as having pointer alignment, 4 or 8 byte, rather than inheriting alignment from the element type. - FFI: support for bitfields in the face of alignment and packing. - FFI: bug in internal type cloning function, leaving the type descriptor structure pointing to the original type as its "self", manifesting itself as a wrong result from something like (alignof foo.bar) when foo.bar has a cloned type (e.g. by the align operator). - UTF8: Incredibly, a bug was found: the UTF-8 decoder was silently eating an incomplete character at the end of the input, instead of treating the incomplete sequence as bad bytes, to be mapped into the U+DCxx range. Thus there were binary strings which were not preserved in the decode -> encode round trip. - OOP: Fixed an out-of-bounds stack access in the struct type initialization code which deals with suppressing redundant initializations of repeated multiple-inheritance bases. This more readily affects big-endian systems: showed upon PPC64. TXR 275 2022-05-10 Features - Architecture support: RISC-V and Loongarch (64 bit). - Hashing: new group-map function: group-reduce with built-in map pass. - Lib: new isecp function: test whether two sequences intersect without calculating intersectin. - FFI: - intmax-t and uintmax-t types. - new str-s, bstr-s and wstr-s types for receiving foreign string without freeing its memory. - after a FFI call, the arguments are processed for reverse data flows and memory clean-up in reverse order. - with above two features, strtol can be wrapped in FFI, including the error-reporting char **end pointer. - Loading: new *load-search-dirs* variable. - default search directory list includes sysrooted lib dir Bugs - configure: minor escaping corner cases in in production of ./reconfigure script. - compiler: package-related bug in file-compilation, reported by Paul A. Patience. - load: regression: do not try adding suffixes to a path which exists; try the given path before anything else - Reported by Paul A. Patience. - sh, run, open-command and open-process now flush *stdout* in situations when it makes sense, standard output is ordered between the subprocesses and the parent. - listener: Ctrl-Z issue when txr is one of multiple processes in a job control process group. TXR 274 2022-02-24 Features - Configure/Build: - 'make clean-c" now cleans the C object files without removing .tlo files. - complementary to "make clean-tlo". - experimental, not tested support for configuring 64 bit time_t on 32 bit Glibc. - CPPFLAGS (C preprocessor flags) variable noticed and used now. - TXR now supports building with -fsanitize=undefined option. - you must specify it yourself via platform-flags, etc. - configure detects it and puts #define HAVE_UBSAN 1 into config.h - FFI: - new feature: enumed bitfield type combination now works. - Doc: - numerous documentation fixes. - Lib: - cptr-int: allow full unsigned range, so pointers can be specified as unsigned integers, or using negative signed values also. - New copy-cptr function; copy copies cptr objects. - New nandf and norf functions. - New function random-sample for one-pass reservoir sampling of a sequence. - load: supports catenated .tlo files now - cat-files: new function for catenating files, like POSIX cat. - find-max uses generic iteration. - new find-max-key function. - new partition-if function. - new list-builder method oust - also local function in build macro - Getopts: - various improvements. - opt-help function/method split up into several. - Macros: - New etypecase macro. - New nand and nor macros and functions. - opip now allows embedded (ap ...) and so on. - Compiler: - new optimizations. - TXR Pattern Language: - new function match-fboundp for testing whether a symbol has a binding as a pattern function. - Expander: - new @,expr hack: quasiquote generates (sys:var ...) or (sys:expr ...) based on type of substituted value. - macro-time is no longer a special operator, but a macro. - Listener: - Hack: Ctrl-V Ctrl-J now inserts CR (i.e. new line in multiline mode) rather than a LF. - Good for people used to inserting line breaks in GNU Readline. - Improvement in method completion. - getopts: - Numerous improvements, mainly in area of help generation. - Autoloading: - More nuanced implementation with multiple symbol namespaces, reduces spurious loading of modules not actually used. Bugs - TXR Pattern Language: - bug fixed in @(freeform) - involves bugfix in lazy-str-get-trailing-list function. - filtering now throws when there is an invalid filter, due to a fix in the filter-string-tree function. - Parser: - bug: carriage returns in JSON not tolerated. - Configure/Build - fixed broken file offset bits detection, resulting in no large file support on 32 bit Glibc platforms (regression since 244). - fixed broken syntax in unwind.h causing build to break if CONFIG_DEBUG_SUPPORT disabled. - Macros: - sum-each, mul-each: handle no vars case. - typecase: return nil from formless clauses. - fix broken :key parameters. - Lib: - carray: allow t and floating 0 in sub and replace. - carray-replace: two overrun bugs. - separate: wrong return value when seq is nil. - time structure: added missing wday and yday slots. - Listener: - bug handling comments in plain mode. - issue handling Ctrl-C in plain mode. - Structural Pattern Matching: - quasiliteral match wrongly allowing loose prefix matching. - `@{nil #/regex/}` wrongly throwing exception. - Command Line: - -Dvar now binds var to empty string rather than t. - this t was some thing inadvertently introduced in 2014. - -Dvar=foo=bar (value containing equal sign) works. - -Dx,y,z now diagnosed. - Vim Syntax Files: - improvement in handling multi-line string literals. - Search trees: - fixed two array underruns found by ubsan. - both situations work reliably by fluke in unfixed code due the memory cell below the array reliably being zero bits. - PRNG: - undefined behavior (32 bit shift of 32 bit value) in random function. - termios: - variables cmspar and crtscts had wrong values on 32 bits due to overflow in initialization. - General: - numerous numeric conversion issues identified by ubsan were addressed in various places in the code base. - Missing autoload for *in-compilation-unit* caused loading problem for compiled code making use of with-compilation-unit. TXR 273 2021-12-28 Features - compiler: - new jump optimizations. - register compacting optimization: greatly reduces stack use, especially of complex functions, and improves cache locality. - pattern language: - @{var /regex/} changes: - regex no longer ignored when var already has binding - text is extracted with regex, then compared to variable - @{var (fun ..)} changes: - (fun ...) now processed in vertical mode if sole item in line. - variable captures lines skipped over in vertical processing. - (fun ...) no longer ignored when var already has binding. - fun executed like in unbound case. - text that would be bound to variable is compared to existing value. - structural pattern matching: - in quasiliteral patterns, @{var #/regex/} can specify bound variable now: - matches text in same way as unbound case - matched text must then match content of var - lib: - new functions: tuples*, rot, nrot, subq, subq, subqual, subst, pairlis. - hash tables: - use 64 bit hash on 64 bit platforms, rather than 32 bit. - search trees: - new function tree-count, and length/len works on trees. - duplicate keys supported: - tree, tree-insert, tree-insert-node have optional argument for allowing duplicates. - tree-delete-specific-node allows specific node to be removed, when removing by key is ambiguous. - priority queue support: - tree-min, tree-min-node, tree-del-min, tree-del-min-node - oop/structs: - new feature: application-defined struct clause macros. - new: :delegate and :mass-delegate clause macros for generating delegate method boilerplate with minimal code. Bugs - compiler: - fix non-working (compile '(lambda ...)). - buffers: - file-get-buf and command-get-buf use unbuffered I/O to read the exact number of bytes into the buffer, avoiding reading more bytes than requested. - case macros (mainly affecting casequal): - fixed 2017 regression causing a key value like ((a b c)), which is the single key (a b c), to be wrongly converted into a list of three keys. - each-match, each-prod, sum-each family of macros: - documented and added missing anonymous block - maprodo: spurious non-nil return value issue. - interpreter: bug in interpreting optional parameters, present in original implementation from 2014 (absent in compiler). - iteration: gc stability problem in iter-begin and iter-reset. - define-accessor: broken argument handling. - less/greater: gaping bug, vectors not supported. TXR 272 2021-11-11 Features: - path manipulation: - new path-equal function for comparing paths. - pic macro: - support for digit-separating commas. - support for (...) notation for negative values. - FFI: - internal improvements and minor optimizations. - more ergonomic handling of carray, cptr passed by pointer. - compiler: - now diagnoses constant expressions that throw. - improved elimination of wasteful jmp instructions. - minor new optimization eliminating a wasteful register copy. - PRNG: - new random-float-incl function: like random-float but the range is [0, 1] rather than [0, 1). - syntactic places: - new read-once accessor for caching a place so that it is read only once even by place mutating operators which access it more than once. - ifa macro semantics adjusted to take advantage of read-once. Bugs: - FFI: - broken range checks in enum types. - bad format calls in enum error handling code. - math: - bad edge cases in 64 bit conversion (affecting 32-bit platforms). - path manipulation: - rel-path bugfixes for native Windows. - printer: - cases where fallback package syms are wrongly printed without package prefix. - compiler: - ordering issue in load-time. - incorrect algebraic transformation of (- a b c ...) minus forms. - incorrect code generation when compiling catch forms. - top-level lambdas no longer captured into D registers: - not strictly a bug, but undesirable behavior that crept in when lambda lifting by load-time was introduced. - syntax: - broken #; syntax for first element of list. - listener: - bug causing incomplete auto-loading of modules during Tab completion. - structural pattern matching: - unquoted quasiliteral patterns now work. - less function: - crash when arguments are symbolic and the right one is nil. - other inconsistent, incorrect behavior for some combinations of symbolic arguments. TXR 271 2021-10-05 Features: - load: - new *load-hooks*: defer exeucution to load finish time. - used via push-after-load and pop-after-load - libtags.txr: script for generating extra tags to jump to TXR's C code using Lisp symbols. - lib: - delcons function: destructively remove indicated cons from list. - improvements in string-extend. - set-mask and clear-mask macros: shorter code working with masks. - new module for quantile estimation: see quantile function. - summing and producing variations of each operator: - sum-each-prod, mul-each-prod, sum-each-prod*, mul-each-prod* - path-search: semantics changes. - path access test functions now use read uid/gid rather than effective uid/gid. - new replace-env function. - new *child-env* variable for specifying environment for executed process images. - FFI: - socklen-t type now defined. - ffi macro now generates load-time form to avoid repeated invocation of FFI type compiler. - new cptr-carray function, inverse of carray-cptr. - exceptions: - system exceptions now store errno in exception message. - see string-get-code in doc. - sockets: - socket options now supported via new sock-opt and set-sock-opt. - exception now thrown if socket call fails. - compiler: - one small optimization improvement and internal improvements. - awk: - new :fields feature to give fields names and type conversions. Bugs: - compiler: - regression in calculation of output path of compiled files. - random perturbation in code generation due to dependency on hash table order in an optimization routine. - poll: array from alloca passed to free. - sockets: bug in sock-peer assignment place. - hash: gc problem in copy-hash function. - sequence iteration: gc problem. - maprodo: problm with single-list argument. TXR 270 2021-08-30 Features - open-file now supports "x" mode for exclusive create, contributed by Paul A. Patience. Bugs - sequence iteration: - garbage collection corruption was discovered via experimentation with string ranges (new feature in 269). - caused by not properly doing the counter-generation object mutations. - Bug is not new: affects bignum ranges, oop sequences. - iterating ranges that go from fixnum to bignum now allowed. - open-file: - "+" mode was behaving like "r" and not "r+" (Paul A. Patience) - "w+", "m+" and "a+" refused to create file (Paul A. Patience) TXR 269 2021-08-28 Twelfth Anniversary Edition Features - networking: - getaddrinfo now implements ai-canonname flag. - system interface: - mmap function now supported, integrated with carray. - mprotect, msync, madvise are there. - structural pattern matching: - new match and match-ecase macros for irrefutable matching. - basic Lisp: - new ecase family of macros. - sequences/iterables: - string ranges like "AAA".."ZZZ" are now iterable. - iterators (objects from iter-begin) are iterable. - sub function allows iterables. - FFI: - carray-pun function allows displacement - improved support for big/little endian types: - more efficient when matches local endian - aligned reads and writes transfer word at a time. - PRNG: - new random-buf function for obtaining a block of pseudo-random bytes. - doc improvements. - build/port: - builds on FreeBSD; test cases pass. - new --no-full-repl option builds trimmed-down listener that supports only plain mode editing, not requiring termios. Bugs - int-str: bug stripping 0x unconditionally regardess of radix argument. - format: leading sign state leaking into subsequent conversions. - ffi: deffi: broken support for variadic functions. - random: bug with modulus that is multiple of 32 bits, found on PPC64. - open-file: "+" mode must be equivalent to "r+" not "r". TXR 268 2021-08-07 Features - subtypep: arguments can now be struct type objects returned by find-struct-type, not only type symbols. - JSON: - new *read-bad-json* dynamic variable, enabling tolerance for trailing commas in JSON arrays and objects. - OOP: - syntactic infelicity in new* and lnew* operators addressed. - streams: - close-stream now replays return value if called redudnantly. - get-lines/lazy-stream-cons now have optional parameter controlling whether the implicit close-stream can throw. - listener: plain mode - handles multi-line expressions - prints prompts if stdin is tty - prompts can be turned on with :prompt-on - banner is suppressed when stdin isn't tty; more usable in pipes. - TXR Pattern Language: - @(eof) now takes an optional argument which can bind the exit status of the input source. Useful for process pipes. Bugs - gc: aborts caused by incorrectness in several object-copying functions. - correct diagnostic name in remql function. - listener: plain mode (txr -n, or input is not a terminal) now handles multi-line expressions. - build: musl fix for socket.c: need . - streams: incorrect argument defaulting of second arg of close-stream TXR 267 2021-07-26 Features - system interface: - new getrlimit, setrlimit functions - random numbers: - buffer objects can be used as random seeds now - build: - txr.c now recompiled if build_id changes. - PDF build is now reproducible even if ghostscript and groff don't have patches for this. - tags.tl: - now supports --emacs argument for Emacs-style tags, thanks to Paul A. Patience. - other improvements - hashes: - Hashes now support both and-semantics and or-semantics for tables that have both weak keys and values. - and-semantics means both key and value must be unreachable for the hash entry to disappear. - or-semantics means the entry disappears if just the key or the value is unreachable. Bugs - compat: - glaring bug fixed going back more than 150 versions. - certain effects of the -C compatibility option not having their documented effect. - caused by referencing the opt_compat variable in global initialization functions, at which time opt_compat is always zero due to -C not having been processed yet. - Test case fixes for missing /bin/sh situation exemplified by Guix build environment. - op: - weirdness in handling nested do (do do do ...) fixed - hashes: - fixed TXR 235 regression in weak processing, causing entries to spuriously disappear from weak hash tables that are only referenced by other weak hash tables. - fixed incorrect recalculation of hash table counts of weak hash tables during garbage collection. - carray: - missing type checking in a couple of functions, creating opportunity for trivial crash. TXR 266 2021-07-12 Features - built-in macros and special operators are now subject to more rigorous syntax checking during the macro-expansion walk. - improvements in error reporting - built-in macros use compiler-like error reporting now. - Lisp files executed from command line rather than loaded, ditto. - running make tests out of an editor now takes you to the error line. - improvements in doc function, and OpenBSD support. - type system overhauled to disallow structs that clash with built-in types. - new function called separate contributed by Paul A. Patience. - combines keep-if and remove-if semantics - new path-manipulation-related functions trim-short-suffix, trim-long-suffix and add-suffix. - new build-id feature: optional string that can be inserted into TXR at build time, displayed by txr --build-id. Bugs - non-functional chmod.tl test case fixed, thanks to Paul A. Patience. - regex: argument defaulting problem in regex-compile. - *stderr* stream is now sanely reset during unhandled exception processing. - new steps taken to prevent runaway recursion in exception processing. - streams: - close-stream function refuses to close stderr. (previously refused only stdin and stdout.) - put-char, put-line: lack of type checking on stream argument. - bug in with-resources problem fixed, reported by Paul A. Patience. - doc ignores BROWSER variable if it is empty. TXR 265 2021-07-04 Features - requirements change in new long-suffix and short-suffix functions: - dot is now part of suffix. - leading dot is not a suffix delimiter: e.g. .bashrc is not a suffix. - trailing path separators ignored, like in base-name. - regex: optimization function exposed. - constantp function now recognizes more kinds of expressions: - (+ 1 (* 3 4)) is constantp, as is (symacrolet ((a (+ 2 2))) (* b 3)). - doc function - now handles situations in which xdg-open blocks until browser exits. - now reacts to BROWSER variable, and if xdg-open is not found, falls back on the first of a long list of possible browsers. - filesystem interface: - path-cat function is now variadic: (path-cat "a" "b" "c" ...). - new path-search function, searches for an executable by name in path, defaulting to the system's PATH. - sequences: - new find-true function; like find-if, but returns the true value that the predicate produces, rather than the item from the sequence. - I/O streams: - argument defaulting tightened; functions no longer treat a nil value for the stream argument as a missing argument. - stack limit: - minimum limit now imposed when the system's stack limit is too low, rather than disabling the mechanism. - stack limit is now always on, even if we don't obtain a value from the system or that value indicates that there is no limit. - documentation infrastructure: - improvements from Paul A. Patience integrated. - doc workflow catches more kinds of problems. - listener: empty EDITOR variable now treated as missing. Bugs - build: regression in separate-directory build. - parser: regression: not working with byacc. - compiler: a number of bugs in inline lambda implementation. - op: subtle bug in do operator; code refactored. - base-name function: problem with empty suffix. - listener: end-of-line/buffer visual glitch in selection. - trie: bugs in regex-from-trie function, now covered by tests. - regex: print/read consistency problem printing n-ary operators. - doc: *doc-url* variable not special, as documented. - getopts: throwing sys:opt-error instead of usr:opt-error. - command line: lack of robustness in -b option fixed. - documentation: numerous fixes. - packages: find-symbol was behaving identically to find-symbol-fb. - signals: itimer-prof variable misspelled as itimer-prov. - search trees: documented tnodep function now actually exists. - stack limit: fix crash when system stack limit is RLIM_INFINITY. TXR 264 2021-06-25 Features - system interface: - TXR no longer relies on popen for open-command. - glob function accepts multiple pattern arguments and uses multiple calls to the C function with GLOB_APPEND. - parser: - parsing Lisp or JSON from a string now produces error if there is any trailing material in the string. - paths: - new functions short-suffix and long-suffix for robustly extracting the suffixes/extensions of path names. - lib: - new functions cxr and cyr for traversing cons-cell structures using a car/cdr path binary-coded in an integer. - mismatch/rmismatch better optimized for strings - starts-with and ends-with use these. - structural pattern matching: - new looping macros while-match, while-match-case, while-true-match-case. - parser: - no longer wastefully allocates dynamic string when scanning a floating-point token. - tests: - target-installable test cases are now relocatable (can be installed at any path) due to a small improvement in the run.sh script. - program-wide: - share/txr/stdlib moved to stdlib. - type mismatches when a string is expected now give function name in error diagnostic. - stack overflow protection is introduced: - in key places, TXR detects whether the stack pointer is over a predetermined limit and throws a stack-overflow exception. - controlled by set-stack-limit function. Bugs - fixed wrong result from (rmismatch #() ()) and (rmismatch "" ()). TXR 263 2021-06-17 Features - New macro named flow, providing the syntactic sugar for using an opip function on a value. - I/O: - the *stdnull* stream lazily attaches to /dev/null if fileno is invoked on it - formatted printing: - format: new precision modifier - for zero instead of plus sign. - pic macro: takes advantage of format work to generate better code. - subprocesses: - some file descriptor saving-restoring manipulations moved into child process (in open-process, open-subprocess, run) - diagnostic for situation when *stdout*, *stdin* or *stderr* are redirected to something that cannot produce a file descriptor. - match-fun/txr-if - documented that input can be a stream - documented that input can be a single string - txr-case: - if input is a stream, it is now converted to a lazy list of lines, so that the txr-case construct effectively backtracks over the data as it tries successive cases. - command-line - new --noprofile option to invoke listener without processing ~/.txr_profile file. Bugs - format: numeric handling maintenance - poor behaviors identified and revised. - requirements clarified. - cemented in test cases. - exceptions: - unwind dynamic environment when tracing unhandled exception - solves problem when exception goes off while *stderr* is redirected. - subprocesses: - diagnostic for situation when *stdout*, *stdin* or *stderr* are redirected to something that cannot produce a file descriptor. - macros: - fixed TXR 191 regression in defsymacro: expanding the replacement form before associating it with the symbol, rather than taking as-is. - quasiliterals: - fixed issue arising when a macro invoked as a @(...) expression in a quasiliteral expands to a non-string atom. - math: - forbid dubious inequality comparisons like (< 1 "abc") which became unintentionally allowed due to numbers being iterable. TXR 262 2021-06-11 Features - structural pattern matching: - new feature: quasiquote matching. - JSON: - improved escaping of JSON output for safe embedding in