Hi all, 

TXR 281 [1] is out. 

Binaries in usual location [2]. 

In this release, an ancient issue is addressed. 

When we use lazy lists that take data from I/O streams, those lazy lists hang on to open streams. The stream won't close until either (1) the lazy list reads it to the end, (2) the lazy list is reclaimed as garbage or (3) something explicitly closes the stream behind the lazy list's back. 

Because of this, if a program creates many lazy lists of lines against an open stream, and just lets them become garbage, this can lead to the process running out of open file descriptors: it's the old problem of using garbage collection to manage non-memory external resources. 

This situation can occur in the use of function like file-get-lines and get-lines. It also occurs in the TXR Pattern Language, which uses lazy lists of strings to scan input sources. 

TXR 281 provides solutions for these situations. There is a new macro called close-lazy-streams. You simply wrap this macro around code that uses lazy lists over streams. When the macro terminates, all streams that were associated with lazy lists inside the enclosed code (dynamically, not lexically!) are closed. 

A test case for it looks like this:

(dotimes (i 20000) 
  (close-lazy-streams 
    (file-get-lines self-path))) 

The close-lazy-streams will close the stream opened by file-get-lines, so this executes without running out of file descriptors. 

In the pattern language, input sources are now closed when the pattern matching is done with their list (either by backtracking away or successfully matching everything).  For instance if we have this: 

@(block)
@(next "bigfile")
@(data captured-lazy-list)
@(end) 

At the end of the block, the stream opened for "bigfile" will now be closed. However, the captured-lazy-list variable is still in scope. This means that the captured lazy list will run into an error when we try to use it. For this relatively rare situation, there is a new :noclose keyword supported by @(next). 

@(block)
@(next "bigfile" :noclose)
@(data captured-lazy-list)
@(end) 

In summary: opt-in solution in Lisp, opt-out solution in the pattern language. 

Cheers .... 

 

Links:
------
[1] https://www.kylheku.com/cgit/txr/tree/RELNOTES?id=txr-281
[2] https://www.kylheku.com/txr-downloads/txr-281/