New: stack overflow protection.

* configure: detect getrlimit, producing HAVE_RLIMIT in
config.h.

* eval.c (do_eval, do_expand): Call gc_stack_check inline
function to check stack pointer against limit.

* gc.c (gc_stack_bottom): Static becomes extern, so inline
function in gc.h can refer to it.
(gc_stack_limit): New global variable.
(gc_init): If we have rlimit, then probe RLIMIT_STACK.
If the stack is sufficiently large, then enable the stack
overflow protection, which kicks in when the stack pointer
appears to be within a certain percentage of the limit.
(set_stack_limit, get_stack_limit): New static functions.
(gc_late_init): Register set-stack-limit and get-stack-limit
intrinsics.
(gc_stack_overflow): New function.

* gc.h (gc_stack_bottom, gc_stack_limit, gc_stack_overflow):
Declared.
(gc_stack_check): New inline function.

* lib.c (stack_overflow_s): New symbol variable.
(obj_print_impl): Call gc_stack_check to protect recursive
printing againts overflow.

* lib.h (stack_overflow_s): Declared.

* unwind.c (uw_init): Register stack-overflow symbol as a an
exception symbol subtyped from error.
(uw_unwind_to_exit_point): When dealing with an unhandled
exception, turn off the stack limit, so we can print the
messages without triggering it in a loop.

* vm.c (vm_execute_closure, vm_funcall_common): Insert
gc_stack_check to the top of the execution of every VM
function.

* txr.1: Documented.

* share/txr/stdlib/doc-syms.tl: Updated.

1 files changed, 3 insertions, 0 deletions

diff --git a/vm.c b/vm.c
index da06d156..f3aada5c 100644
--- a/vm.c
+++ b/vm.c
@@ -1115,6 +1115,8 @@ val vm_execute_closure(val fun, struct args *args)
   cnum ix = 0;
   vm_word_t argw = 0;
 
+  gc_stack_check();
+
   vm_reset(&vm, vd, dspl, vc->nlvl - 1, vc->ip);
 
   vm.dspl = coerce(struct vm_env *, frame + vc->nreg);
@@ -1174,6 +1176,7 @@ val vm_execute_closure(val fun, struct args *args)
   int frsz = vd->nlvl * 2 + vc->nreg;                                        \
   val *frame = coerce(val *, zalloca(sizeof *frame * frsz));                 \
   struct vm_env *dspl = coerce(struct vm_env *, frame + vc->nreg);           \
+  gc_stack_check();                                                          \
   vm_reset(&vm, vd, dspl, vc->nlvl - 1, vc->ip);                             \
   vm.dspl = coerce(struct vm_env *, frame + vc->nreg);                       \
   frame[0] = nil;                                                            \