Syntax: allow separator commas in numeric tokens.

* parser.l (remove_char): New static function.
(DIGSEP, XDIGSEP, NUMSEP, FLOSEP, XNUMSEP, ONUMSEP,
BNUMSEP, ONUM, BNUM): New named lex patterns.
(FLODOT): Use DIGSEP instead of DIG.
(ONUM): Use ODIG instead of [0-7].
(BNUM): Use BDIG instead of [0-1].
(grammar): New rule for producing NUMBER from decimal
token with commas based on BNUMSEP instead of BNUM.
This is a copy and paste so that the BNUM rule doesn't
deal with the comma removal, not to slow it down.
For the octal, binary and hex, we just switch to
BNUMSEP, ONUMSEP and XNUMSEP, so they all go through
one case.
Floating point numbers are also handled with a copy
pasted case using FLOSEP.

* tests/012/syntax.tl: New test cases.

* txr.1: Documented.

* genvim.txr (alpha-noe, digsep, hexsep, octsep, binsep): New
variables.
(txr_pnum, txr_xnum, txr_onum, txr_bnum, txr_num): Integrate
separating commas. Some bugs fixed in txr_num, some simplifications,
better txr_badnum pattern.

* lex.yy.c.shipped: Updated.

1 files changed, 27 insertions, 0 deletions

diff --git a/tests/012/syntax.tl b/tests/012/syntax.tl
index 36894610..d2fab599 100644
--- a/tests/012/syntax.tl
+++ b/tests/012/syntax.tl
@@ -36,3 +36,30 @@
    '(#; .abc 1) (1)
    '(0 #; .abc 1) (0 1)
    '(0 #; .abc) (0))
+
+(mtest
+  '(-,1) (- (sys:unquote 1))
+  1,2 12
+  1,,2 12
+  1,,,2 12
+  1,2,3 1,2,3
+  -0,1 -1
+  '(1,a) (1 (sys:unquote a)))
+
+(mtest
+  (read "#x,ff") :error
+  (read "#o,1") :error
+  (read "#b,1") :error
+  '(#xff,ff,z) (65535 (sys:unquote z))
+  '(#xff,ff,a) (1048570))
+
+(mtest
+  #xff,ff 65535
+  #o7,7,7 511
+  #b1101,1110 #xDE)
+
+(mtest
+  1,234,567.890,123E13 1234567.890123E13
+  '(1.234,e+12) (1.234 (sys:unquote e+12))
+  '(1.,234) (1.0 (sys:unquote 234)))
+