Home My Page Projects Code Snippets Project Openings SML/NJ Bugs
Summary Activity Tracker Lists

[#63] Comments not lexed correctly in mlyacc grammar files

Date:
2011-01-19 00:46
Priority:
3
State:
Closed
Submitted by:
Bug Submitter (webuser)
Assigned to:
Nobody (None)
Machine Architecture:
None
Operating System:
Generic Unix
Component:
ML-Yacc
Resolution:
Fixed
Severity:
Cosmetic
OS Version:
SML/NJ Version:
110.72
Keywords:
mlyacc lexing
URL:
Transcript (of reproduction):
Source (for reproduction):
As above.
Summary:
Comments not lexed correctly in mlyacc grammar files

Detailed description
Comments are not permitted in code snippets:

e.g.,

nonterminal : TOK1 nonterminal2 TOK2 (some code (* comment *))

fails.

Comments are not really lexed as such in the prefix. The following both fail:

(*
this is comment where the comment start is immediate followed by a newline
*)
open foo
%%

as well as

(* lets put a special string in my comment %% *)
maniacal laughter
Fix:
changeset: 7714:9a0d5c89f1e4
user: Michael Norrish michael.norrish@nicta.com.au
date: Wed Jan 12 16:02:09 2011 +1100
files: c-parser/tools/mlyacc/src/yacc.lex
description:
Fix bug in comment-handling in lexer for mlyaccs input language.

In particular, if a comment was successfully lexed, it returned a BOGUS_VALUE
token, which then messed up the higher-level parsing. This was due to
the use of the idiom

continue() before YYBEGIN SOMESTATE

making the return value of the continue the return value for the lexer at
that point.

This bug prevented the use of comments in code sections entirely.

It was somewhat masked in the prefix section by the fact that most
supposed comments there were never lexed as such anyway. This could
be observed by a file that included %% in a comment in the prefix
section.


diff --git a/c-parser/tools/mlyacc/src/yacc.lex b/c-parser/tools/mlyacc/src/yacc.lex
--- a/c-parser/tools/mlyacc/src/yacc.lex
+++ b/c-parser/tools/mlyacc/src/yacc.lex
@@ -71,11 +71,11 @@
qualid ={id}.;
%%
INITIAL(* = (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
- continue() before YYBEGIN INITIAL);
+ continue(); YYBEGIN INITIAL; continue());
A(* = (YYBEGIN EMPTYCOMMENT; commentLevel := 1; continue());
CODE(* = (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
- continue() before YYBEGIN CODE);
-INITIAL[^%\n]+ = (Add yytext; continue());
+ continue(); YYBEGIN CODE; continue());
+INITIAL[^(%\n]+ = (Add yytext; continue());
INITIAL%% = (YYBEGIN A; HEADER (concat (rev (!text)),!lineno,!lineno));
INITIAL,CODE,COMMENT,F,EMPTYCOMMENT\n = (Add yytext; inc lineno; continue());
INITIAL. = (Add yytext; continue());


Submitted via web form by Michael Norrish michael.norrish@nicta.com.au

Comments:

Message  ↓
Date: 2011-04-08 17:04
Sender: John Reppy

Added suggested fix for 110.73.

Attached Files:

Changes

Field Old Value Date By
close_date2011-11-26 15:502011-11-26 15:50jhr
ResolutionNone2011-11-26 15:50jhr
status_idOpen2011-04-08 17:04jhr
close_date2011-04-08 17:042011-04-08 17:04jhr