bugfix: fixnum crackdown.

The purpose of this commit is to address certain situations in
which code is wrongly relying on a cnum value being in the
fixnum range (NUM_MIN to NUM_MAX), so that num_fast can safely
be used on it.

One wrong pattern is that c_num is applied to some Lisp value,
and that value (or one derived from it arithmetically) is then
passed to num_fast. The problem is that c_num succeeds on
integers outside of the fixnum range.  Some bignum values
convert to a cnum successfully. Thus either num has to be used
instead of num_fast, or else the original c_num attempt must
be replaced with something that will fail if the original
value isn't a fixnum. (In the latter case, any arithmetic on
the fixnum cannot produce value outside of that range).

* buf.c (buf_put_bytes): The size argument here is not
guaranteed to be in fixnum range: use num.

* combi.c (perm_init_common): Throw if the sequence length
isn't a fixnum. Thus the num_fast in perm_while_fun is
correct, since the ci value is bounded by k, which is bounded
by n.

* hash.c (hash_grow): Remove dubious assertion which aborts
the run-time if the hash table doubling overflows.  Simply
don't allow the modulus to grow beyond NUM_MAX.  If doubling
it makes it larger than NUM_MAX, then just don't grow the
table. We need the modulus to be in fixnum range, so that
uses of num_fast on the modulus value elsewhere are correct.
(group_by, group_reduce): Use c_fixnum rather than c_num to
extract a value that is later assumed to be a fixnum.

* lib.c (c_fixnum): New function.
(nreverse, reverse, remove_if, less, window_map_list,
sort_vec, unique): Use c_fixnum rather than c_num to extract a
value that is later assumed to be a fixnum.
(string_extend): Use c_fixnum rather than c_num to extract a
value that is later assumed to be a fixnum. Cap the string
allocation size to fixnum range rather than INT_PTR_MAX.
(cmp_str): The wcscmp function could return values outside
of the fixnum range, so we must use num, not num_fast.

* lib.h (c_fixnum): Declared.

1 files changed, 9 insertions, 4 deletions

diff --git a/hash.c b/hash.c
index c7603f55..7198c2dc 100644
--- a/hash.c
+++ b/hash.c
@@ -539,9 +539,12 @@ static void hash_grow(struct hash *h, val hash)
 {
   cnum i;
   cnum new_modulus = 2 * h->modulus;
-  val new_table = vector(num_fast(new_modulus), nil);
+  val new_table;
 
-  bug_unless (new_modulus > h->modulus);
+  if (new_modulus > NUM_MAX)
+    return;
+
+  new_table = vector(num_fast(new_modulus), nil);
 
   for (i = 0; i < h->modulus; i++) {
     val conses = vecref(h->table, num_fast(i));
@@ -1096,12 +1099,13 @@ val hash_list(val keys, struct args *hashv_args)
 
 val group_by(val func, val seq, struct args *hashv_args)
 {
+  val self = lit("group-by");
   val hash = hashv(hashv_args);
 
   if (vectorp(seq)) {
     cnum i, len;
 
-    for (i = 0, len = c_num(length(seq)); i < len; i++) {
+    for (i = 0, len = c_fixnum(length(seq), self); i < len; i++) {
       val v = vecref(seq, num_fast(i));
       pushhash(hash, funcall1(func, v), v);
     }
@@ -1126,12 +1130,13 @@ val group_by(val func, val seq, struct args *hashv_args)
 val group_reduce(val hash, val by_fun, val reduce_fun, val seq,
                  val initval, val filter_fun)
 {
+  val self = lit("group-reduce");
   initval = default_null_arg(initval);
 
   if (vectorp(seq)) {
     cnum i, len;
 
-    for (i = 0, len = c_num(length(seq)); i < len; i++) {
+    for (i = 0, len = c_fixnum(length(seq), self); i < len; i++) {
       val v = vecref(seq, num_fast(i));
       val key = funcall1(by_fun, v);
       val new_p;