Home My Page Projects Code Snippets Project Openings SML/NJ
Summary Activity Forums Tracker Lists Tasks Docs Surveys News SCM Files

SCM Repository

[smlnj] Annotation of /sml/trunk/tests/coresml/doc/testsuit.asc
ViewVC logotype

Annotation of /sml/trunk/tests/coresml/doc/testsuit.asc

Parent Directory Parent Directory | Revision Log Revision Log


Revision 548 - (view) (download)

1 : dbm 548
2 :    
3 :    
4 :    
5 :     Test suite for the Core Language of Standard ML
6 :    
7 :    
8 :    
9 :    
10 :     Michael Vium, C 908944
11 :    
12 :     and
13 :    
14 :     Sten Schmidt, C 908841
15 :    
16 :    
17 :     Department of Computer Science
18 :     Technical University of Denmark
19 :    
20 :    
21 :    
22 :    
23 :    
24 :    
25 :    
26 :    
27 :    
28 :    
29 :    
30 :    
31 :    
32 :    
33 :    
34 :    
35 :    
36 :    
37 :    
38 :    
39 :    
40 :    
41 :    
42 :    
43 :    
44 :     September 1, 1994
45 :    
46 :     Contents
47 :    
48 :    
49 :    
50 :    
51 :    
52 :     Acknowledgement 1
53 :    
54 :     1.0 Introduction 2
55 :    
56 :     2.0 What is tested 3
57 :    
58 :     3.0 Approach 4
59 :    
60 :     3.1 Testing systematically 4
61 :    
62 :     3.2 Naming files systematically 4
63 :    
64 :     3.3 Test suite programs 5
65 :    
66 :     3.4 Testing rule 17 6
67 :    
68 :     3.4.1 Test of the valbinding 7
69 :    
70 :     3.4.2 Test of Closure of the variable environment VE7
71 :    
72 :     4.0 Using the test suite 12
73 :    
74 :     5.0 Conclusion 13
75 :    
76 :     Bibliography 15
77 :    
78 :     A Appendix: Which rules are used in which files 16
79 :    
80 :     B Appendix: Test results from four SML implementations22
81 :    
82 :     Acknowledgement
83 :    
84 :    
85 :    
86 :    
87 :    
88 :     We would like to thank Peter Sestoft at the Department of Computer
89 :     Science, Technical University of Denmark for giving us the
90 :     opportunity to work on this project. Also, we would like to thank
91 :     him for his assistance in the development of this test suite.
92 :     1.0 Introduction
93 :    
94 :    
95 :    
96 :    
97 :    
98 :     This document describes the development of a test suite that can be
99 :     used to test Standard ML implementations. A precise and formal
100 :     definition of Standard ML is given in The Definition of Standard
101 :     ML written by Milner, Tofte, and Harper available from MIT Press.
102 :     The idea with this project was systematically to develop a test suite
103 :     based on the definitions present in the Definition. The definition of
104 :     the Standard ML programming language is different from the
105 :     definitions of most other programming languages þ it mathematically
106 :     describes not only the grammar but also the meaning of the
107 :     language. This is why it seemed reasonable to try to construct a
108 :     test suite based on the definitions available, rule by rule. Because
109 :     of the time limitations imposed on the project, this test suite is
110 :     limited to testing the static semantics for the Core Language of
111 :     Standard ML. The dynamic semantics for the Core Language and the
112 :     Module Language is not covered by this work.
113 :    
114 :     The test suite is formed by a collection of small Standard ML
115 :     programs. Each program is constructed to test a rule or a part of a
116 :     rule, e.g. a side condition. Also each program is accompanied by a
117 :     description of how the compiler is expected to react to it. For
118 :     example, the program might fail because of a type error. On the
119 :     other hand, the program might succeed in which case its expected
120 :     behaviour must be documented.
121 :    
122 :     An exhaustive test is impossible, so care is needed in selecting
123 :     cases that are most likely to reveal errors. Some program
124 :     constructions are more complicated than others, and in general more
125 :     complicated constructions are somewhat more likely to fool a compiler
126 :     than simple ones. Nevertheless, the use of this test suite on four
127 :     different Standard ML implementations showed us, that even simple
128 :     illegal constructs were accepted by some implementations (for
129 :     example, two implementations accepted free imperative type variables
130 :     at top level).
131 :    
132 :     Section 2 of this document describes exactly which parts of the
133 :     Definition are tested by this test suite and which are not. Section 3
134 :     describes the approach used in the development. It contains
135 :     information regarding testing systematically, naming files
136 :     systematically and the test suite programs in general. Also the
137 :     testing of rule 17 is outlined as an example of how rules are tested
138 :     systematically. Section 4 describes how to use the test suite. Finally,
139 :     section 5 is dedicated to the conclusion. Note, that it is not the
140 :     intention with this document to describe the development of each
141 :     program in this test suite in detail. Neither does it try to explain
142 :     definitions nor rules in the Definition. Anyone interested in the
143 :     definitions of Standard ML is advised to take a look at the
144 :     Definition or [2]. 2.0 What is tested
145 :    
146 :    
147 :    
148 :    
149 :    
150 :     As stated in the introduction this test suite is limited to testing the
151 :     static semantics for the Core Language of Standard ML. The dynamic
152 :     semantics for the Core Language and the Module Language are not
153 :     covered by this work.
154 :    
155 :    
156 :     This test suite tests
157 :    
158 :     þ rule 100 and the static semantics for the Core Language (the
159 :     rules from 1 to 52, rule 23 excluded) [chapter 4].
160 :    
161 :     þ the syntactic restrictions [page 9].
162 :    
163 :     þ the derived forms [page 67-68].
164 :    
165 :     þ the first point in the further restrictions [page 30].
166 :    
167 :    
168 :     This test suite does not test
169 :    
170 :     þ the Module Language.
171 :    
172 :     þ the dynamic semantics for the Core Language [chapter 6]
173 :    
174 :     þ the initial static basis [Appendix C].
175 :    
176 :     þ the initial dynamic basis [Appendix D] thoroughly.
177 :    
178 :     þ the grammar of the Core Language [page 8-9] and [Appendix
179 :     B] systematically or thoroughly.
180 :    
181 :     þ whether the compiler warns regarding match exhaustiveness
182 :     and redundant patterns as demanded in the second point in
183 :     the further restrictions [page 30].
184 :    
185 :     þ the scoping of explicit type variables in valbindings
186 :     completely (the test programs R017A-AC.ML and R017K-FL.ML
187 :     may however indicate whether the compiler treats the scoping
188 :     correctly).
189 :    
190 :     3.0 Approach
191 :    
192 :    
193 :    
194 :    
195 :    
196 :     3.1 Testing systematically
197 :    
198 :     The rules in the Definition are tested one by one and the test
199 :     programs are kept as simple as possible. This makes it easier to
200 :     ensure that the rules are tested in all possible instances. It also
201 :     minimizes the chance of problems in one declaration affecting on the
202 :     testing of another.
203 :    
204 :     þ for each rule all the side conditions are tested separately in
205 :     different test programs.
206 :    
207 :     þ for each side condition there is a program testing that the
208 :     compiler will fail to elaborate when the side condition does not
209 :     hold.
210 :    
211 :     þ all instances where a side condition does not hold are tested
212 :     separately.
213 :    
214 :     þ the test programs in the present test suite are generally kept
215 :     very simple except for the test programs for the more
216 :     essential rules in ML.
217 :    
218 :     For each rule there is also one or more programs that test the rule
219 :     in general. These are the programs that requires intuition to
220 :     construct. The better knowledge of SML implementations and
221 :     problems herein, the better test programs one can conceive. They
222 :     should be designed so they are most likely to fool a compiler,
223 :     although they must remain correct according to the Definition.
224 :    
225 :     Naturally, the biggest problem testing compilers systematically is
226 :     that not all side conditions can be tested that easily. For instance,
227 :     some of the side conditions make demands on how the compiler
228 :     should react internally. This can not be tested directly in a
229 :     program. However, if the compiler does not comply with these
230 :     conditions it could have unpredictable consequences in the
231 :     elaboration of the following declarations. Normally, if the compiler
232 :     treats complicated examples correctly, chances are that it will treat
233 :     all programs correctly with respect to this specific side condition.
234 :     This is the philosophy behind this test suite when testing internal
235 :     behavior of the compiler.
236 :    
237 :    
238 :     3.2 Naming files systematically
239 :    
240 :     For convenience the test programs are named the following way:
241 :    
242 :     {1 char category} + {3 char rule/restriction/derived form number} +
243 :     {1 char serial number A-Z} + "-" + {2 char compiler reaction} +
244 :     ".ML"
245 :    
246 :     The test programs are divided into four categories:
247 :    
248 :     R Rules. These are the main programs in the test suite. They
249 :     test the rules as given in the Definition. The number in the
250 :     filename equals the rule number.
251 :    
252 :     S Syntactic restrictions. These programs test that the compiler
253 :     respects these extra restrictions. The number in the filename
254 :     equals the point number [page 9].
255 :    
256 :     D Derived Forms. These programs test that the compiler
257 :     implements all the derived forms as mentioned in [Appendix A].
258 :    
259 :    
260 :     M Miscellaneous. These files test various details in SML. Se notes
261 :     in files for further information.
262 :    
263 :     The reactions indicated in the filenames can be either AC or FL
264 :     meaning that the compiler should accept or fail respectively.
265 :    
266 :     Examples:
267 :    
268 :     R017B-FL.ML: this program is the second program that tests rule 17
269 :     and the compiler should refuse to elaborate it.
270 :    
271 :     S002A-AC.ML: this program is the first program that tests the
272 :     second point in the syntactic restrictions and the compiler should
273 :     succeed in elaborating it.
274 :    
275 :    
276 :     3.3 Test suite programs
277 :    
278 :     Some considerations have been made in order to make it as easy as
279 :     possible to test compilers with this test suite. All programs in this
280 :     test suite are made the same way:
281 :    
282 :     þ They are all as small as possible.
283 :    
284 :     þ The programs that are meant to fail will fail in the last
285 :     declaration of the program. This ensures that a deliberate
286 :     error does not influence the elaboration of other tests.
287 :    
288 :     þ In all value declarations resulting in boolean variables, the
289 :     variables are named test, test1, test2 and so forth. The
290 :     programs are all constructed so the boolean variables should
291 :     elaborate to true. This way, it will be obvious, that if a
292 :     variable evaluates to false there is an error in the compiler.
293 :    
294 :     þ The programs in the rules category list the rules in (* *) that
295 :     are used in each line. This makes the programs somewhat
296 :     harder to read. However, it enables the reader to ensure for
297 :     himself that the program really tests what it is meant to test.
298 :     It also makes it possible to see which programs uses which
299 :     rules.
300 :    
301 :     þ All the programs in the test suite contain a comment at the
302 :     end showing the expected output from the SML compiler. Note
303 :     however, that different SML implementations print output in
304 :     different ways. This is why the expected output is only
305 :     advisory. It cannot be exact for all SML implementations.
306 :    
307 :     The test programs for the derived forms are constructed in a
308 :     somewhat different way. More than one derived form are tested in
309 :     the same program. This means that most of these programs are quite
310 :     long compared to the programs in the other three categories. In
311 :     these programs all declarations resulting in non-boolean variables
312 :     are hidden in local declarations. This is done to cut down on the
313 :     material needed to be examined in order to see if the compiler does
314 :     it right. As usual all the boolean variables should evaluate to true.
315 :     Furthermore, in the last declaration of the programs in this
316 :     category a variable named alltrue should evaluate to true if all the
317 :     other boolean variables are true. This way, only the variable alltrue
318 :     has to be inspected to see if the compiler elaborates and evaluates
319 :     the derived forms correctly.
320 :    
321 :    
322 :     3.4 Testing rule 17
323 :    
324 :     In this section we will describe the test programs used to test rule
325 :     17. This description is also an example of how one can systematically
326 :     construct programs to test a given rule. These programs have not
327 :     been chosen at random of course. Rule 17 is interesting because it
328 :     concerns the value binding which is fundamental to SML. The fact
329 :     that it is the most difficult rule to test also makes it the obvious
330 :     choice of rule to describe thoroughly. It is beyond the scope of this
331 :     document to describe the development of all the programs in this
332 :     test suite in such detail.
333 :    
334 :     Rule 17 concerns value declarations such as:
335 :    
336 :     val pat = exp
337 :    
338 :     The rule has two side conditions:
339 :    
340 :     1. VE'=ClosC,valbindVE
341 :     2. U ï tyvars VE' = 
342 :    
343 :     Each one of these side conditions is tested separately. The first test
344 :     program R017A-AC.ML tests valbindings in general. The next
345 :     programs from
346 :     R017B-FL.ML to R17I-FL.ML test the closure of the variable
347 :     environment VE. The program R017J-FL.ML tests the second side
348 :     condition. Finally, the program R017K-FL.ML tests the scoping of
349 :     explicit type variables. Note that in the valbindings in the programs
350 :     R017A-AC.ML and R017K-FL.ML the explicit type variables scoped
351 :     here have been listed in brackets {}.
352 :    
353 :    
354 :     3.4.1 Test of the valbinding
355 :    
356 :     R0017A-AC.ML:
357 :    
358 :     val{} x = (let val{'a} Id1 : 'a -> 'a = fn z => z in Id1 Id1 end,
359 :     let val{'a} Id2 : 'a -> 'a = fn z => z in Id2 Id2 end);
360 :    
361 :     The first test program is taken directly from [page 20]. It tests the
362 :     scoping of explicit type variables. Since 'a is scoped locally at the
363 :     second value declaration, let val Id1: 'a -> 'a = fn z => z, it should
364 :     result in the variable environment VE' = {Id þ þ'a.'a 'a}. For the
365 :     same reason the third value declaration, let val Id2 : 'a -> 'a = fn z =>
366 :     z, should result in the variable environment VE' = {Id þ þ'b.'b 'b}.
367 :     Hence the applications Id1 Id1 and Id2 Id2 should elaborate.
368 :    
369 :    
370 :     3.4.2 Test of closure of the variable environment VE
371 :    
372 :     The next programs test the different alternatives the SML compiler
373 :     has to contemplate when closing the variable environment in rule 17.
374 :     As stated [page 21], the compiler should determine whether the
375 :     expression exp is expansive or non-expansive and do the closure
376 :     operation accordingly. The type variables can be either applicative
377 :     or imperative and either free or not free in the context C. This
378 :     results in the following eight possibilities and test programs.
379 :    
380 :    
381 :     test program exp type variable type variable quantification
382 :     in C
383 :     -----------------------------------------------------------------------------
384 :     R017B-FL.ML non-expansive applicative free no
385 :     R017C-AC.ML non-expansive applicative not free yes
386 :     R017D-AC.ML non-expansive imperative not free yes
387 :     R017E-FL.ML non-expansive imperative free no
388 :     R017F-FL.ML expansive applicative free no
389 :     R017G-AC.ML expansive applicative not free yes
390 :     R017H-FL.ML expansive imperative not free no
391 :     R017I-FL.ML expansive imperative free no
392 :    
393 :    
394 :    
395 :     R017B-FL.ML:
396 :    
397 :     val f = fn x => let
398 :     val y = x
399 :     in
400 :     not y andalso y = 5
401 :     end;
402 :    
403 :     The valbinding, val y = x, is elaborated in the context C = C0 + {x þ
404 :     'a} where C0 is the initial context. But since 'a is free in the context
405 :     it should not be quantified in this value declaration. Therefore this
406 :     value declaration should result in the variable environment VE' = {y
407 :     þ 'a 'a}. Now 'a can be unified with bool in the expression not y
408 :     but not with int in the same expression as well. Thus the compiler
409 :     should fail to elaborate this program.
410 :    
411 :    
412 :     R017C-AC.ML:
413 :    
414 :     val test = let
415 :     val f = fn z => z
416 :     in
417 :     (f 3 = 3) andalso f true
418 :     end;
419 :    
420 :     The valbinding, val f = fn z => z, is elaborated in the context C0. But
421 :     since no type variables are free in the context and exp is non-
422 :     expansive, 'a should be quantified in this value declaration.
423 :     Therefore this value declaration should result in the variable
424 :     environment VE' = {f þ þ'a.'a 'a}. Now 'a can be unified with int in
425 :     the expression f 3 = 3 and with bool in the expression f true.
426 :    
427 :    
428 :     R017D-AC.ML:
429 :    
430 :     let
431 :     val f = fn x => ref x
432 :     in
433 :     f 3 ; f true
434 :     end;
435 :    
436 :     The valbinding, val f = fn x => ref x, is elaborated in the context C0.
437 :     But since no type variables are free in the context and the
438 :     expression in the valbinding, fn x => ref x, is non-expansive, 'a should
439 :     be quantified. Therefore the value declaration should result in the
440 :     variable environment VE' = {f þ þ'a.'a 'a ref}. Now 'a can be
441 :     unified with int in the expression f 3 and with bool in the
442 :     expression f true.
443 :    
444 :    
445 :     R017E-FL.ML:
446 :    
447 :     val f = fn x => let
448 :     val y = ref x
449 :     in
450 :     not(!y) andalso !y = 5
451 :     end;
452 :    
453 :     The valbinding, val y = ref x, is elaborated in the context C0 + {x þ
454 :     'a}. But since the type variable for x, 'a, is free in the context 'a
455 :     should not be quantified in this value declaration. Therefore this
456 :     value declaration should result in the variable environment VE' = {y
457 :     þ 'a ref}. Now 'a can be unified with bool in the expression not(!y)
458 :     but not with int in the expression !y = 5. Thus the compiler should
459 :     fail to elaborate this program.
460 :    
461 :    
462 :     R017F-FL.ML:
463 :    
464 :     fn x => let
465 :     val g = (fn y => x) 5
466 :     in
467 :     (g = 5.5) andalso g
468 :     end;
469 :    
470 :     The valbinding, val g = (fn y => x) 5, is elaborated in the context C0 +
471 :     {x þ 'a}. But since the type variable for x, 'a, is free in the context
472 :     C, 'a should not be quantified in this value declaration. Therefore
473 :     the value declaration should result in the variable environment VE'
474 :     = {g þ 'a}. Now 'a can be unified with real in the expression g = 5.5
475 :     but not with bool in the expression g. Thus the compiler should fail
476 :     to elaborate this program.
477 :    
478 :    
479 :     R017G-AC.ML:
480 :    
481 :     val test = let
482 :     val r = (fn x => (x, ref [])) (fn y => y)
483 :     in
484 :     let
485 :     val (f,y) = r
486 :     in
487 :     f 3 = 3 andalso f true andalso ( y := [1]; [1] = !y)
488 :     end
489 :     end;
490 :    
491 :     The valbinding, val r = (fn x => (x, ref [])) (fn y => y), is elaborated in
492 :     the context C0. But since no type variables are free in the context
493 :     C, the applicative type variable 'a should be quantified. Therefore
494 :     this value declaration should give the variable environment VE' = {r
495 :     þ þ'a.('a 'a) * 'b list ref }. Now 'a can be unified with int in the
496 :     expression f 3 = 3 and also with bool in the expression f true.
497 :    
498 :    
499 :     R017H-FL.ML:
500 :    
501 :     val test = let
502 :     val r = (fn x => (x, ref [])) (fn y => y)
503 :     in
504 :     let
505 :     val (f,y) = r
506 :     in
507 :     f 3 = 3
508 :     andalso f true
509 :     andalso (y := [1]; [1] = !y)
510 :     andalso (y := [true]; [true] = !y)
511 :     end
512 :     end;
513 :    
514 :     The valbinding, val r = (fn x => (x, ref [])) (fn y => y), is elaborated in
515 :     the context C0. But since the type of ref [] contains an imperative
516 :     type variable 'b and the expression is expansive 'b should not be
517 :     quantified in this value declaration. Therefore this value declaration
518 :     should result in the variable environment VE' = {r þ þ'a.('a 'a) *
519 :     'b list ref }. The program should not elaborate since 'b can not be
520 :     unified with int list in the expression y := [1] and also with bool
521 :     list in the expression y := [true]. Thus the compiler should fail to
522 :     elaborate this program.
523 :    
524 :    
525 :     R017I-FL.ML:
526 :    
527 :     val f = fn x => let
528 :     val y = ref x
529 :     in
530 :     y := 1 ; y := true
531 :     end;
532 :    
533 :     The valbinding, val y = ref x, is elaborated in the context C0 + {x þ
534 :     'a}. But since 'a is free in this context 'a should not be quantified
535 :     in this value declaration. Therefore the value declaration should
536 :     result in the variable environment VE' = {y þ 'a ref}. The program
537 :     should not elaborate since 'a cannot be unified with int in the
538 :     expression y := 1 and also with bool in the expression y := true.
539 :     Thus the compiler should fail to elaborate this program.
540 :    
541 :    
542 :     R017J-FL.ML:
543 :    
544 :     fn x => let
545 :     val{'a} y : 'a = x
546 :     in
547 :     y
548 :     end;
549 :    
550 :     In this program 'a is scoped at the val declaration, val{'a} y : 'a = x.
551 :     The declaration is elaborated in the context C0 + {x þ 'b}. But since
552 :     the value declaration sets y equal to x the two type variables, 'a
553 :     and 'b, must also be the same. This breaks the second side
554 :     condition of rule 17 (U ï tyvars VE' = ). Thus the program should
555 :     not elaborate.
556 :    
557 :     R017K-FL.ML:
558 :    
559 :     val{'a} x = (let val{} Id : 'a -> 'a = fn z => z in Id Id end,
560 :     fn z => z : 'a);
561 :    
562 :     This program is taken from [page 20]. It tests the scoping of
563 :     explicit type variables. The type variable 'a is scoped at the outer
564 :     value declaration. This means that the valbinding, val{} Id : 'a -> 'a =
565 :     fn z => z, elaborates into VE = {Id þ 'a 'a}. This way Id can not
566 :     occur in two different instantiations in the expression Id Id. Thus
567 :     the program should not elaborate.
568 :    
569 :    
570 :     4.0 Using the test suite
571 :    
572 :    
573 :    
574 :    
575 :    
576 :     The testing of a compiler with this test suite can not be made
577 :     automatically. However, the work is made somewhat easier by the
578 :     accessory files provided with it. For testing compilers running
579 :     under MS-DOS the following batch files are provided:
580 :    
581 :     TEST_ALL.BAT Runs all the test programs on the compiler and
582 :     directs the output into a subdirectory called
583 :     LOGFILES.
584 :    
585 :     TEST_ONE.BAT Runs one test program on a compiler. The user
586 :     should change the contents of this file with the
587 :     name of the compiler to be tested. Used by
588 :     TEST_ALL.BAT.
589 :    
590 :     SHOW_ALL.BAT Shows all the test programs one by one with the
591 :     result from the compiler. This makes it possible to
592 :     compare the actual results with the expected
593 :     results to see if the compiler has elaborated and
594 :     evaluated the programs correctly.
595 :    
596 :     SHOW_ONE.BAT Used by SHOW_ALL.BAT to show one test program
597 :     along with the output from the compiler.
598 :    
599 :     For testing compilers running under UNIX the script TEST_ALL.S is
600 :     provided. The command:
601 :    
602 :     sh TEST_ALL.S > compiler.log
603 :    
604 :     runs the compiler with all the test programs and stores all the
605 :     output into the file compiler.log.
606 :    
607 :     Please note that it is very important that the output from the
608 :     compiler is carefully examined. It should be checked that the output
609 :     from the compiler matches the expected results. This does not mean
610 :     that the compiler should give the same results syntactically but
611 :     semantically. The test results from the failing programs should be
612 :     examined closely to see if an error message really is a result of a
613 :     deliberate error in a program.
614 :    
615 :     5.0 Conclusion
616 :    
617 :    
618 :    
619 :    
620 :    
621 :     As stated in the introduction it is not possible to prove that a
622 :     Standard ML implementation, or any other program for that matter,
623 :     works as intended to do by using it on test data. For example, one
624 :     cannot ensure that a every new datatype in a SML program is really
625 :     assigned a new internal type in the SML implementation (rule 29).
626 :     Also, one cannot test the internal interpretation of a SML program.
627 :     This is why, for example, it is not possible to ensure that derived
628 :     forms are interpreted correctly. One can create a number of test
629 :     programs containing derived forms and examine if they behave as
630 :     expected. But one cannot really check that the derived forms are
631 :     interpreted correctly internally in the SML implementation. Even
632 :     though the compiler handles the examples correctly one will never
633 :     know if the compiler will handle all other instances of the derived
634 :     forms correctly as well.
635 :    
636 :     So is the approach used in this project appropriate, e.g. is
637 :     systematical testing based on the rules from the Definition the best
638 :     way to test a SML implementation? We think so for the following
639 :     reasons:
640 :    
641 :     þ SML is very precisely mathematically defined which in itself
642 :     makes it possible to use a systematic approach.
643 :    
644 :     þ by systematically testing rule by rule chances are that one
645 :     gets to test things that might otherwise not be thought of.
646 :    
647 :     þ by testing one thing at a time in separate programs one
648 :     ensures that a failure to elaborate or evaluate a declaration is
649 :     not a side effect of a previous declaration.
650 :    
651 :     þ we have found a number of errors in different SML
652 :     implementations.
653 :    
654 :     The question then is if we have found more errors than we would
655 :     by testing at random. We are convinced that we actually have. The
656 :     arguments are the same as above plus the fact that this test suite
657 :     makes it very easy to compare expected results with actual results.
658 :     Also by constructing test programs at random chances are that
659 :     different programs might actually test the same thing. One would
660 :     have to check the Definition to see what a program actually tested.
661 :     This approach could lead to much more work.
662 :    
663 :     Having said this about the systematical approach we should add, as
664 :     stated before, that intuition was also a part of the development of
665 :     the programs in this test suite, e.g. the rules in the Definition
666 :     cannot be translated directly into programs. The construction of a
667 :     program for a specific rule or a side condition sometimes required a
668 :     lot of effort.
669 :    
670 :     Finally, we would like to reemphasize that an exhaustive test is
671 :     impossible using test data. If a SML implementation elaborates (and
672 :     evaluates) all programs in this test suite correctly it may of course
673 :     still contain errors.
674 :    
675 :     Bibliography
676 :    
677 :    
678 :    
679 :    
680 :    
681 :     [1] Robin Milner, Mads Tofte, and Robin Harper. The Definition of
682 :     Standard ML. MIT Press, 1990.
683 :    
684 :     [2] Robin Milner and Mads Tofte. Commentary on Standard ML. MIT
685 :     Press, 1991.
686 :    
687 :     [3] Chris Reade. Elements of functional programming. Addison-
688 :     Wesley Publishers Ltd, 1989.
689 :    
690 :     [4] Robert Harper. Introduction to Standard ML. School of
691 :     Computer Science, Carnegie Mellon University, 1990.
692 :    
693 :    
694 :     A. Which rules are used in which files
695 :    
696 :    
697 :    
698 :    
699 :    
700 :     The following list shows which rules are used in which files. Each of
701 :     the rules 1-22, 24-52 is followed by a list of the files in which a
702 :     rule is used at least once. As it can be seen from the list, all the
703 :     rules 1-22, 24-52 are used in the test suite.
704 :    
705 :    
706 :     Rule 1:
707 :     R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006A-AC.ML, R007A-AC.ML, R008A-AC.ML,
708 :     R010A-AC.ML, R010B-FL.ML, R011A-AC.ML, R011B-FL.ML, R011D-FL.ML, R012A-AC.ML,
709 :     R012B-FL.ML, R012C-FL.ML, R013A-AC.ML, R014A-AC.ML, R015A-AC.ML, R015B-FL.ML,
710 :     R015C-FL.ML, R016A-AC.ML, R016B-AC.ML, R017B-FL.ML, R017C-AC.ML, R017E-FL.ML,
711 :     R017F-FL.ML, R017G-AC.ML, R017I-FL.ML, R018A-AC.ML, R020A-AC.ML, R020B-FL.ML,
712 :     R020C-FL.ML, R021A-AC.ML, R021B-FL.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML,
713 :     R024A-AC.ML, R025A-AC.ML, R026A-FL.ML, R026B-FL.ML, R027A-AC.ML, R027B-AC.ML,
714 :     R027C-FL.ML, R028A-AC.ML, R029A-AC.ML, R029B-AC.ML, R030A-AC.ML, R031A-AC.ML,
715 :     R031B-FL.ML, R031C-FL.ML, R034A-AC.ML, R034B-AC.ML, R035A-AC.ML, R038A-AC.ML,
716 :     R039A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML, R044B-FL.ML,
717 :     R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R048A-AC.ML, R049A-AC.ML,
718 :     R049B-FL.ML,
719 :    
720 :     Rule 2:
721 :     R002A-FL.ML, R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006A-AC.ML, R006B-FL.ML,
722 :     R006C-FL.ML, R007A-AC.ML, R008A-AC.ML, R010A-AC.ML, R010B-FL.ML, R011A-AC.ML,
723 :     R011E-FL.ML, R013A-AC.ML, R014A-AC.ML, R015A-AC.ML, R015B-FL.ML, R015C-FL.ML,
724 :     R016B-AC.ML, R017A-AC.ML, R017B-FL.ML, R017C-AC.ML, R017D-AC.ML, R017E-FL.ML,
725 :     R017F-FL.ML, R017G-AC.ML, R017H-FL.ML, R017I-FL.ML, R017J-FL.ML, R017K-FL.ML,
726 :     R018A-AC.ML, R019A-AC.ML, R019B-AC.ML, R019C-FL.ML, R019D-AC.ML, R020A-AC.ML,
727 :     R020B-FL.ML, R020C-FL.ML, R020D-AC.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML,
728 :     R024A-AC.ML, R025A-AC.ML, R026B-FL.ML, R027A-AC.ML, R027B-AC.ML, R027C-FL.ML,
729 :     R027D-FL.ML, R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R029C-FL.ML, R031A-AC.ML,
730 :     R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R036A-AC.ML, R038A-AC.ML, R039A-AC.ML,
731 :     R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML, R044B-FL.ML, R045A-AC.ML,
732 :     R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML,
733 :    
734 :     Rule 3:
735 :     R003A-FL.ML, R005A-AC.ML, R005B-FL.ML, R006A-AC.ML, R006B-FL.ML, R006C-FL.ML,
736 :     R008A-AC.ML, R010A-AC.ML, R011A-AC.ML, R011B-FL.ML, R011C-FL.ML, R013A-AC.ML,
737 :     R017C-AC.ML, R017D-AC.ML, R017E-FL.ML, R017G-AC.ML, R017H-FL.ML, R017I-FL.ML,
738 :     R019A-AC.ML, R019B-AC.ML, R019C-FL.ML, R019D-AC.ML, R020A-AC.ML, R020B-FL.ML,
739 :     R020C-FL.ML, R020D-AC.ML, R022A-AC.ML, R022B-AC.ML, R025B-FL.ML, R027D-FL.ML,
740 :     R028B-AC.ML, R029A-AC.ML, R029B-AC.ML, R029C-FL.ML, R029D-FL.ML, R030A-AC.ML,
741 :     R031A-AC.ML, R031B-FL.ML, R031C-FL.ML, R034A-AC.ML, R034C-AC.ML, R035A-AC.ML,
742 :     R036A-AC.ML, R038A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML,
743 :     R044B-FL.ML, R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R048A-AC.ML,
744 :     R049A-AC.ML, R049B-FL.ML, R050A-AC.ML, R052A-AC.ML, R100A-FL.ML,
745 :    
746 :     Rule 4:
747 :     R013A-AC.ML, R013B-FL.ML, R021A-AC.ML, R044A-AC.ML,
748 :    
749 :     Rule 5:
750 :     R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006B-FL.ML, R006C-FL.ML, R007A-AC.ML,
751 :     R008A-AC.ML, R010A-AC.ML, R010B-FL.ML, R011A-AC.ML, R011B-FL.ML, R013A-AC.ML,
752 :     R015B-FL.ML, R017A-AC.ML, R017B-FL.ML, R017C-AC.ML, R017E-FL.ML, R017F-FL.ML,
753 :     R017G-AC.ML, R017H-FL.ML, R017I-FL.ML, R017K-FL.ML, R018A-AC.ML, R019A-AC.ML,
754 :     R019B-AC.ML, R019C-FL.ML, R019D-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML,
755 :     R020D-AC.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R025A-AC.ML, R027A-AC.ML,
756 :     R027B-AC.ML, R027C-FL.ML, R027D-FL.ML, R028A-AC.ML, R029A-AC.ML, R029B-AC.ML,
757 :     R029C-FL.ML, R030A-AC.ML, R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R035A-AC.ML,
758 :     R038A-AC.ML, R039A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML,
759 :     R044B-FL.ML, R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R048A-AC.ML,
760 :     R049A-AC.ML, R049B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML,
761 :    
762 :     Rule 6:
763 :     R006A-AC.ML, R006B-FL.ML, R006C-FL.ML, R010B-FL.ML, R017A-AC.ML, R017B-FL.ML,
764 :     R017C-AC.ML, R017D-AC.ML, R017E-FL.ML, R017F-FL.ML, R017G-AC.ML, R017H-FL.ML,
765 :     R017I-FL.ML, R017J-FL.ML, R019B-AC.ML, R019C-FL.ML, R019D-AC.ML, R024A-AC.ML,
766 :     R025B-FL.ML, R029B-AC.ML, R031B-FL.ML, R031C-FL.ML,
767 :    
768 :     Rule 7:
769 :     R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R007A-AC.ML, R008A-AC.ML, R010B-FL.ML,
770 :     R011A-AC.ML, R011E-FL.ML, R013A-AC.ML, R014A-AC.ML, R017C-AC.ML, R017F-FL.ML,
771 :     R017G-AC.ML, R017H-FL.ML, R019A-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML,
772 :     R025A-AC.ML, R027A-AC.ML, R027B-AC.ML, R027C-FL.ML, R031A-AC.ML, R031B-FL.ML,
773 :    
774 :     Rule 8:
775 :     R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006B-FL.ML, R006C-FL.ML, R007A-AC.ML,
776 :     R008A-AC.ML, R010A-AC.ML, R010B-FL.ML, R011A-AC.ML, R011B-FL.ML, R013A-AC.ML,
777 :     R015B-FL.ML, R017A-AC.ML, R017B-FL.ML, R017C-AC.ML, R017E-FL.ML, R017F-FL.ML,
778 :     R017G-AC.ML, R017H-FL.ML, R017I-FL.ML, R017K-FL.ML, R018A-AC.ML, R019A-AC.ML,
779 :     R019B-AC.ML, R019C-FL.ML, R019D-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML,
780 :     R020D-AC.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R025A-AC.ML, R027A-AC.ML,
781 :     R027B-AC.ML, R027C-FL.ML, R027D-FL.ML, R028A-AC.ML, R029A-AC.ML, R029B-AC.ML,
782 :     R029C-FL.ML, R030A-AC.ML, R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R035A-AC.ML,
783 :     R038A-AC.ML, R039A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML,
784 :     R044B-FL.ML, R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R048A-AC.ML,
785 :     R049A-AC.ML, R049B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML,
786 :    
787 :     Rule 9:
788 :     R002A-FL.ML, R003A-FL.ML, R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006A-AC.ML,
789 :     R006B-FL.ML, R006C-FL.ML, R007A-AC.ML, R008A-AC.ML, R010A-AC.ML, R010B-FL.ML,
790 :     R011A-AC.ML, R011B-FL.ML, R011C-FL.ML, R011D-FL.ML, R011E-FL.ML, R012A-AC.ML,
791 :     R012B-FL.ML, R012C-FL.ML, R013A-AC.ML, R013B-FL.ML, R014A-AC.ML, R015A-AC.ML,
792 :     R015B-FL.ML, R015C-FL.ML, R016A-AC.ML, R016B-AC.ML, R017A-AC.ML, R017B-FL.ML,
793 :     R017C-AC.ML, R017D-AC.ML, R017E-FL.ML, R017F-FL.ML, R017G-AC.ML, R017H-FL.ML,
794 :     R017I-FL.ML, R017J-FL.ML, R017K-FL.ML, R018A-AC.ML, R019A-AC.ML, R019B-AC.ML,
795 :     R019C-FL.ML, R019D-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R020D-AC.ML,
796 :     R021A-AC.ML, R021B-FL.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R024A-AC.ML,
797 :     R025A-AC.ML, R025B-FL.ML, R026A-FL.ML, R026B-FL.ML, R027A-AC.ML, R027B-AC.ML,
798 :     R027C-FL.ML, R027D-FL.ML, R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R029A-AC.ML,
799 :     R029B-AC.ML, R029C-FL.ML, R029D-FL.ML, R030A-AC.ML, R031A-AC.ML, R031B-FL.ML,
800 :     R031C-FL.ML, R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R035A-AC.ML, R036A-AC.ML,
801 :     R038A-AC.ML, R039A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML,
802 :     R044B-FL.ML, R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R048A-AC.ML,
803 :     R049A-AC.ML, R049B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML, R100A-FL.ML,
804 :    
805 :     Rule 10:
806 :     R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006B-FL.ML, R006C-FL.ML, R007A-AC.ML,
807 :     R008A-AC.ML, R010A-AC.ML, R010B-FL.ML, R011A-AC.ML, R013A-AC.ML, R014A-AC.ML,
808 :     R015B-FL.ML, R016B-AC.ML, R017A-AC.ML, R017B-FL.ML, R017C-AC.ML, R017D-AC.ML,
809 :     R017E-FL.ML, R017F-FL.ML, R017G-AC.ML, R017H-FL.ML, R017I-FL.ML, R017K-FL.ML,
810 :     R018A-AC.ML, R019A-AC.ML, R019B-AC.ML, R019C-FL.ML, R019D-AC.ML, R020A-AC.ML,
811 :     R020B-FL.ML, R020C-FL.ML, R020D-AC.ML, R021A-AC.ML, R022A-AC.ML, R022B-AC.ML,
812 :     R022C-FL.ML, R025A-AC.ML, R027A-AC.ML, R027B-AC.ML, R027C-FL.ML, R027D-FL.ML,
813 :     R028A-AC.ML, R028B-AC.ML, R029A-AC.ML, R029B-AC.ML, R029C-FL.ML, R030A-AC.ML,
814 :     R031A-AC.ML, R031B-FL.ML, R031C-FL.ML, R039A-AC.ML, R041A-AC.ML, R043A-AC.ML,
815 :     R043B-FL.ML, R044A-AC.ML, R044B-FL.ML, R100A-FL.ML,
816 :    
817 :     Rule 11:
818 :     R011A-AC.ML, R011B-FL.ML, R011D-FL.ML, R011E-FL.ML, R014A-AC.ML, R016B-AC.ML,
819 :     R017K-FL.ML, R020A-AC.ML, R025B-FL.ML, R048A-AC.ML, R049A-AC.ML, R049B-FL.ML,
820 :    
821 :     Rule 12:
822 :     R012A-AC.ML, R012B-FL.ML, R012C-FL.ML, R013A-AC.ML, R031A-AC.ML, R031B-FL.ML,
823 :    
824 :     Rule 13:
825 :     R013A-AC.ML, R013B-FL.ML, R031A-AC.ML, R031B-FL.ML,
826 :    
827 :     Rule 14:
828 :     R008A-AC.ML, R011E-FL.ML, R014A-AC.ML, R015A-AC.ML, R015B-FL.ML, R015C-FL.ML,
829 :     R016A-AC.ML, R016B-AC.ML, R017A-AC.ML, R017B-FL.ML, R017C-AC.ML, R017D-AC.ML,
830 :     R017E-FL.ML, R017F-FL.ML, R017G-AC.ML, R017H-FL.ML, R017I-FL.ML, R017J-FL.ML,
831 :     R017K-FL.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R022A-AC.ML, R022B-AC.ML,
832 :     R022C-FL.ML, R027A-AC.ML, R027B-AC.ML, R027C-FL.ML, R027D-FL.ML, R028B-AC.ML,
833 :     R034C-AC.ML, R036A-AC.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML,
834 :    
835 :     Rule 15:
836 :     R008A-AC.ML, R011E-FL.ML, R012A-AC.ML, R012B-FL.ML, R012C-FL.ML, R013A-AC.ML,
837 :     R014A-AC.ML, R015A-AC.ML, R015B-FL.ML, R015C-FL.ML, R016A-AC.ML, R016B-AC.ML,
838 :     R017A-AC.ML, R017B-FL.ML, R017C-AC.ML, R017D-AC.ML, R017E-FL.ML, R017F-FL.ML,
839 :     R017G-AC.ML, R017H-FL.ML, R017I-FL.ML, R017J-FL.ML, R017K-FL.ML, R020A-AC.ML,
840 :     R020B-FL.ML, R020C-FL.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R027A-AC.ML,
841 :     R027B-AC.ML, R027C-FL.ML, R027D-FL.ML, R028B-AC.ML, R031A-AC.ML, R031B-FL.ML,
842 :     R034C-AC.ML, R036A-AC.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML,
843 :    
844 :     Rule 16:
845 :     R008A-AC.ML, R011E-FL.ML, R012A-AC.ML, R012B-FL.ML, R012C-FL.ML, R013A-AC.ML,
846 :     R014A-AC.ML, R015A-AC.ML, R015B-FL.ML, R015C-FL.ML, R016A-AC.ML, R016B-AC.ML,
847 :     R017A-AC.ML, R017B-FL.ML, R017C-AC.ML, R017D-AC.ML, R017E-FL.ML, R017F-FL.ML,
848 :     R017G-AC.ML, R017H-FL.ML, R017I-FL.ML, R017J-FL.ML, R017K-FL.ML, R020A-AC.ML,
849 :     R020B-FL.ML, R020C-FL.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R027A-AC.ML,
850 :     R027B-AC.ML, R027C-FL.ML, R027D-FL.ML, R028B-AC.ML, R031A-AC.ML, R031B-FL.ML,
851 :     R034C-AC.ML, R036A-AC.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML,
852 :    
853 :     Rule 17:
854 :     R002A-FL.ML, R003A-FL.ML, R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006A-AC.ML,
855 :     R006B-FL.ML, R006C-FL.ML, R007A-AC.ML, R008A-AC.ML, R010A-AC.ML, R010B-FL.ML,
856 :     R011A-AC.ML, R011B-FL.ML, R011C-FL.ML, R011D-FL.ML, R011E-FL.ML, R012A-AC.ML,
857 :     R012B-FL.ML, R012C-FL.ML, R013A-AC.ML, R013B-FL.ML, R014A-AC.ML, R015A-AC.ML,
858 :     R015B-FL.ML, R015C-FL.ML, R016A-AC.ML, R016B-AC.ML, R017A-AC.ML, R017B-FL.ML,
859 :     R017C-AC.ML, R017D-AC.ML, R017E-FL.ML, R017F-FL.ML, R017G-AC.ML, R017H-FL.ML,
860 :     R017I-FL.ML, R017J-FL.ML, R017K-FL.ML, R018A-AC.ML, R019A-AC.ML, R019B-AC.ML,
861 :     R019C-FL.ML, R019D-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R020D-AC.ML,
862 :     R021A-AC.ML, R021B-FL.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R024A-AC.ML,
863 :     R025A-AC.ML, R025B-FL.ML, R026A-FL.ML, R026B-FL.ML, R027A-AC.ML, R027B-AC.ML,
864 :     R027C-FL.ML, R027D-FL.ML, R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R029A-AC.ML,
865 :     R029B-AC.ML, R029C-FL.ML, R029D-FL.ML, R030A-AC.ML, R031A-AC.ML, R031B-FL.ML,
866 :     R031C-FL.ML, R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R035A-AC.ML, R036A-AC.ML,
867 :     R038A-AC.ML, R039A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML,
868 :     R044B-FL.ML, R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R048A-AC.ML,
869 :     R049A-AC.ML, R049B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML, R100A-FL.ML,
870 :    
871 :     Rule 18:
872 :     R018A-AC.ML, R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R028D-FL.ML, R029A-AC.ML,
873 :     R048A-AC.ML, R049A-AC.ML, R049B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML,
874 :    
875 :     Rule 19:
876 :     R006C-FL.ML, R012C-FL.ML, R013B-FL.ML, R019A-AC.ML, R019B-AC.ML, R019C-FL.ML,
877 :     R019D-AC.ML, R025B-FL.ML, R029A-AC.ML, R029B-AC.ML, R029C-FL.ML, R029D-FL.ML,
878 :     R029E-AC.ML, R030A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML,
879 :    
880 :     Rule 20:
881 :     R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R020D-AC.ML,
882 :    
883 :     Rule 21:
884 :     R012A-AC.ML, R012B-FL.ML, R013A-AC.ML, R021A-AC.ML, R021B-FL.ML, R031A-AC.ML,
885 :     R031B-FL.ML, R031C-FL.ML, R032A-AC.ML, R041A-AC.ML, R044A-AC.ML, R044B-FL.ML,
886 :     R052A-AC.ML,
887 :    
888 :     Rule 22:
889 :     R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R024A-AC.ML,
890 :    
891 :     Rule 24:
892 :     R017D-AC.ML, R017I-FL.ML, R024A-AC.ML,
893 :    
894 :     Rule 25:
895 :     R017D-AC.ML, R017G-AC.ML, R017I-FL.ML, R024A-AC.ML, R025B-FL.ML,
896 :    
897 :     Rule 26:
898 :     R002A-FL.ML, R003A-FL.ML, R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006A-AC.ML,
899 :     R006B-FL.ML, R006C-FL.ML, R007A-AC.ML, R008A-AC.ML, R010A-AC.ML, R010B-FL.ML,
900 :     R011A-AC.ML, R011B-FL.ML, R011C-FL.ML, R011D-FL.ML, R011E-FL.ML, R012A-AC.ML,
901 :     R012B-FL.ML, R012C-FL.ML, R013A-AC.ML, R013B-FL.ML, R014A-AC.ML, R015A-AC.ML,
902 :     R015B-FL.ML, R015C-FL.ML, R016A-AC.ML, R016B-AC.ML, R017A-AC.ML, R017B-FL.ML,
903 :     R017C-AC.ML, R017D-AC.ML, R017E-FL.ML, R017F-FL.ML, R017G-AC.ML, R017H-FL.ML,
904 :     R017I-FL.ML, R017J-FL.ML, R017K-FL.ML, R018A-AC.ML, R019A-AC.ML, R019B-AC.ML,
905 :     R019C-FL.ML, R019D-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R020D-AC.ML,
906 :     R021A-AC.ML, R021B-FL.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R024A-AC.ML,
907 :     R025A-AC.ML, R025B-FL.ML, R026A-FL.ML, R026B-FL.ML, R027A-AC.ML, R027B-AC.ML,
908 :     R027C-FL.ML, R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R029A-AC.ML, R029B-AC.ML,
909 :     R029C-FL.ML, R029D-FL.ML, R030A-AC.ML, R031A-AC.ML, R031B-FL.ML, R031C-FL.ML,
910 :     R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R035A-AC.ML, R036A-AC.ML, R038A-AC.ML,
911 :     R039A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML, R044B-FL.ML,
912 :     R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R048A-AC.ML, R049A-AC.ML,
913 :     R049B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML, R100A-FL.ML,
914 :    
915 :     Rule 27:
916 :     R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R027A-AC.ML, R027D-FL.ML,
917 :    
918 :     Rule 28:
919 :     R018A-AC.ML, R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R028D-FL.ML, R029A-AC.ML,
920 :     R048A-AC.ML, R049A-AC.ML, R049B-FL.ML, R050B-FL.ML, R052A-AC.ML,
921 :    
922 :     Rule 29:
923 :     R006C-FL.ML, R012C-FL.ML, R013B-FL.ML, R019A-AC.ML, R019B-AC.ML, R019C-FL.ML,
924 :     R019D-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R020D-AC.ML, R025B-FL.ML,
925 :     R029A-AC.ML, R029B-AC.ML, R029C-FL.ML, R029D-FL.ML, R029E-AC.ML, R030A-AC.ML,
926 :     R041A-AC.ML, R043A-AC.ML, R043B-FL.ML,
927 :    
928 :     Rule 30:
929 :     R006C-FL.ML, R012C-FL.ML, R013B-FL.ML, R019A-AC.ML, R019B-AC.ML, R019C-FL.ML,
930 :     R019D-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R020D-AC.ML, R029A-AC.ML,
931 :     R029B-AC.ML, R029C-FL.ML, R029D-FL.ML, R029E-AC.ML, R030A-AC.ML, R041A-AC.ML,
932 :     R043A-AC.ML, R043B-FL.ML,
933 :    
934 :     Rule 31:
935 :     R012A-AC.ML, R012B-FL.ML, R013A-AC.ML, R021A-AC.ML, R021B-FL.ML, R031A-AC.ML,
936 :     R031B-FL.ML, R031C-FL.ML, R032A-AC.ML, R041A-AC.ML, R044A-AC.ML, R044B-FL.ML,
937 :     R052A-AC.ML,
938 :    
939 :     Rule 32:
940 :     R032A-AC.ML,
941 :    
942 :     Rule 33:
943 :     R036A-AC.ML, R043A-AC.ML, R043B-FL.ML,
944 :    
945 :     Rule 34:
946 :     R027A-AC.ML, R027B-AC.ML, R027C-FL.ML, R031A-AC.ML, R031B-FL.ML, R034B-AC.ML,
947 :    
948 :     Rule 35:
949 :     R002A-FL.ML, R003A-FL.ML, R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006A-AC.ML,
950 :     R006B-FL.ML, R006C-FL.ML, R007A-AC.ML, R008A-AC.ML, R010A-AC.ML, R010B-FL.ML,
951 :     R011A-AC.ML, R011B-FL.ML, R011C-FL.ML, R011D-FL.ML, R011E-FL.ML, R012A-AC.ML,
952 :     R012B-FL.ML, R012C-FL.ML, R013A-AC.ML, R013B-FL.ML, R014A-AC.ML, R015A-AC.ML,
953 :     R015B-FL.ML, R015C-FL.ML, R016A-AC.ML, R016B-AC.ML, R017A-AC.ML, R017B-FL.ML,
954 :     R017C-AC.ML, R017D-AC.ML, R017E-FL.ML, R017F-FL.ML, R017G-AC.ML, R017H-FL.ML,
955 :     R017I-FL.ML, R017J-FL.ML, R017K-FL.ML, R018A-AC.ML, R019A-AC.ML, R019B-AC.ML,
956 :     R019C-FL.ML, R019D-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R020D-AC.ML,
957 :     R021A-AC.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R024A-AC.ML, R025A-AC.ML,
958 :     R025B-FL.ML, R026B-FL.ML, R027A-AC.ML, R027B-AC.ML, R027C-FL.ML, R027D-FL.ML,
959 :     R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R028D-FL.ML, R029A-AC.ML, R029B-AC.ML,
960 :     R029C-FL.ML, R029D-FL.ML, R030A-AC.ML, R031A-AC.ML, R031B-FL.ML, R031C-FL.ML,
961 :     R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R035A-AC.ML, R036A-AC.ML, R038A-AC.ML,
962 :     R039A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML, R044B-FL.ML,
963 :     R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R048A-AC.ML, R049A-AC.ML,
964 :     R049B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML, R100A-FL.ML,
965 :    
966 :     Rule 36:
967 :     R015A-AC.ML, R015C-FL.ML, R016A-AC.ML, R027D-FL.ML, R031A-AC.ML, R034A-AC.ML,
968 :     R034C-AC.ML, R036A-AC.ML, R044B-FL.ML,
969 :    
970 :     Rule 37:
971 :     R012A-AC.ML, R012B-FL.ML, R012C-FL.ML, R013A-AC.ML, R021B-FL.ML,
972 :    
973 :     Rule 38:
974 :     R008A-AC.ML, R017G-AC.ML, R017H-FL.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML,
975 :     R027D-FL.ML, R031A-AC.ML, R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R036A-AC.ML,
976 :     R038A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML, R044B-FL.ML,
977 :     R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML,
978 :    
979 :     Rule 39:
980 :     R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R027D-FL.ML, R039A-AC.ML,
981 :    
982 :     Rule 40:
983 :     R008A-AC.ML, R038A-AC.ML, R041A-AC.ML, R044B-FL.ML, R045A-AC.ML, R045B-FL.ML,
984 :    
985 :     Rule 41:
986 :     R008A-AC.ML, R017G-AC.ML, R017H-FL.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML,
987 :     R027D-FL.ML, R031A-AC.ML, R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R036A-AC.ML,
988 :     R038A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML, R044B-FL.ML,
989 :     R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML,
990 :    
991 :     Rule 42:
992 :     R002A-FL.ML, R003A-FL.ML, R005A-AC.ML, R005B-FL.ML, R005C-FL.ML, R006A-AC.ML,
993 :     R006B-FL.ML, R006C-FL.ML, R007A-AC.ML, R008A-AC.ML, R010A-AC.ML, R010B-FL.ML,
994 :     R011A-AC.ML, R011B-FL.ML, R011C-FL.ML, R011D-FL.ML, R011E-FL.ML, R012A-AC.ML,
995 :     R012B-FL.ML, R012C-FL.ML, R013A-AC.ML, R013B-FL.ML, R014A-AC.ML, R015A-AC.ML,
996 :     R015B-FL.ML, R015C-FL.ML, R016A-AC.ML, R016B-AC.ML, R017A-AC.ML, R017B-FL.ML,
997 :     R017C-AC.ML, R017D-AC.ML, R017E-FL.ML, R017F-FL.ML, R017G-AC.ML, R017H-FL.ML,
998 :     R017I-FL.ML, R017J-FL.ML, R017K-FL.ML, R018A-AC.ML, R019A-AC.ML, R019B-AC.ML,
999 :     R019C-FL.ML, R019D-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R020D-AC.ML,
1000 :     R021A-AC.ML, R022A-AC.ML, R022B-AC.ML, R022C-FL.ML, R024A-AC.ML, R025A-AC.ML,
1001 :     R025B-FL.ML, R026B-FL.ML, R027A-AC.ML, R027B-AC.ML, R027C-FL.ML, R027D-FL.ML,
1002 :     R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R028D-FL.ML, R029A-AC.ML, R029B-AC.ML,
1003 :     R029C-FL.ML, R029D-FL.ML, R030A-AC.ML, R031A-AC.ML, R031B-FL.ML, R031C-FL.ML,
1004 :     R034A-AC.ML, R034B-AC.ML, R034C-AC.ML, R035A-AC.ML, R036A-AC.ML, R038A-AC.ML,
1005 :     R039A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML, R044B-FL.ML,
1006 :     R045A-AC.ML, R045B-FL.ML, R046A-AC.ML, R046B-FL.ML, R048A-AC.ML, R049A-AC.ML,
1007 :     R049B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML, R100A-FL.ML,
1008 :    
1009 :     Rule 43:
1010 :     R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R027D-FL.ML, R031A-AC.ML, R031B-FL.ML,
1011 :     R036A-AC.ML, R041A-AC.ML, R043A-AC.ML, R043B-FL.ML,
1012 :    
1013 :     Rule 44:
1014 :     R021B-FL.ML, R044A-AC.ML, R044B-FL.ML,
1015 :    
1016 :     Rule 45:
1017 :     R002A-FL.ML, R003A-FL.ML, R011C-FL.ML, R017A-AC.ML, R017J-FL.ML, R017K-FL.ML,
1018 :     R018A-AC.ML, R026A-FL.ML, R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R029A-AC.ML,
1019 :     R029D-FL.ML, R030A-AC.ML, R031A-AC.ML, R043B-FL.ML, R045A-AC.ML, R045B-FL.ML,
1020 :     R046B-FL.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML,
1021 :    
1022 :     Rule 46:
1023 :     R039A-AC.ML, R046A-AC.ML, R046B-FL.ML,
1024 :    
1025 :     Rule 47:
1026 :     R002A-FL.ML, R003A-FL.ML, R011D-FL.ML, R011E-FL.ML, R016B-AC.ML, R017A-AC.ML,
1027 :     R017J-FL.ML, R017K-FL.ML, R028A-AC.ML, R028B-AC.ML, R029A-AC.ML, R029B-AC.ML,
1028 :     R031B-FL.ML, R031C-FL.ML, R049A-AC.ML, R049B-FL.ML, R050A-AC.ML, R050B-FL.ML,
1029 :     R052A-AC.ML,
1030 :    
1031 :     Rule 48:
1032 :     R002A-FL.ML, R003A-FL.ML, R011A-AC.ML, R011B-FL.ML, R019A-AC.ML, R028A-AC.ML,
1033 :     R028B-AC.ML, R029A-AC.ML, R030A-AC.ML, R031A-AC.ML, R032A-AC.ML, R041A-AC.ML,
1034 :     R043A-AC.ML, R043B-FL.ML, R048A-AC.ML, R049A-AC.ML, R049B-FL.ML, R050A-AC.ML,
1035 :     R050B-FL.ML,
1036 :    
1037 :     Rule 49:
1038 :     R002A-FL.ML, R003A-FL.ML, R011A-AC.ML, R011B-FL.ML, R011C-FL.ML, R014A-AC.ML,
1039 :     R018A-AC.ML, R019A-AC.ML, R019B-AC.ML, R019C-FL.ML, R019D-AC.ML, R020A-AC.ML,
1040 :     R020B-FL.ML, R020C-FL.ML, R020D-AC.ML, R021A-AC.ML, R021B-FL.ML, R025B-FL.ML,
1041 :     R026A-FL.ML, R028A-AC.ML, R028B-AC.ML, R028C-AC.ML, R028D-FL.ML, R029A-AC.ML,
1042 :     R029B-AC.ML, R029D-FL.ML, R029E-AC.ML, R030A-AC.ML, R031A-AC.ML, R032A-AC.ML,
1043 :     R041A-AC.ML, R043A-AC.ML, R043B-FL.ML, R044A-AC.ML, R044B-FL.ML, R045A-AC.ML,
1044 :     R045B-FL.ML, R046B-FL.ML, R048A-AC.ML, R049A-AC.ML, R049B-FL.ML, R050A-AC.ML,
1045 :     R050B-FL.ML, R052A-AC.ML,
1046 :    
1047 :     Rule 50:
1048 :     R002A-FL.ML, R003A-FL.ML, R011A-AC.ML, R011E-FL.ML, R017A-AC.ML, R017K-FL.ML,
1049 :     R019C-FL.ML, R028B-AC.ML, R050A-AC.ML, R050B-FL.ML, R052A-AC.ML,
1050 :    
1051 :     Rule 51:
1052 :     R011A-AC.ML, R020A-AC.ML, R020B-FL.ML, R020C-FL.ML, R050A-AC.ML, R050B-FL.ML,
1053 :    
1054 :     Rule 52:
1055 :     R002A-FL.ML, R003A-FL.ML, R011A-AC.ML, R011B-FL.ML, R019A-AC.ML, R028A-AC.ML,
1056 :     R028B-AC.ML, R029A-AC.ML, R030A-AC.ML, R031A-AC.ML, R032A-AC.ML, R041A-AC.ML,
1057 :     R043A-AC.ML, R043B-FL.ML, R048A-AC.ML, R049A-AC.ML, R049B-FL.ML,
1058 :    
1059 :    
1060 :     B. Test results from four SML implementations
1061 :    
1062 :    
1063 :    
1064 :    
1065 :    
1066 :     The test suite was used on the following SML implementations
1067 :    
1068 :     Edinburgh Standard ML, Version 4.1.0, December 13, 1991
1069 :     using EDML3864 V044
1070 :    
1071 :     Micro ML of Umea, Version 0.1, September 24, 1991
1072 :    
1073 :     Moscow SML by Sergei Romanenko, Version 0.04b, August 1994
1074 :    
1075 :     Moscow SML by Sergei Romanenko, Version 1.00, August 1994
1076 :    
1077 :     Standard ML of New Jersey, Version 0.93, February 15, 1993
1078 :    
1079 :     Standard ML of New Jersey, Version 0.93, port no. 2 of May
1080 :     1994, for OS/2.
1081 :    
1082 :     The results of the testing of these SML implementations can be seen
1083 :     below. The list of programs that were elaborated (and evaluated)
1084 :     with results different than those expected contains both AC and FL
1085 :     programs. AC programs appear in the list if they did not elaborate
1086 :     correctly. FL programs appear in the list if they did indeed
1087 :     elaborate even though they should not. The same is true for the list
1088 :     of programs that elaborated differently than expected causing
1089 :     cryptic error messages. However, the test suite programs listed in
1090 :     this category elaborated differently than expected causing a cryptic
1091 :     error message. An example of a cryptic error message is:
1092 :    
1093 :     Can't unify: int*bool
1094 :     With: int*bool*real
1095 :    
1096 :     In expression: val z = (12,(true,3.3)) : int*bool*real;
1097 :    
1098 :     Implying that (true,3.3) has type bool.
1099 :    
1100 :    
1101 :     Edinburgh Standard ML, Version 4.1.0, December 13, 1991
1102 :    
1103 :     The following programs were elaborated differently than expected:
1104 :    
1105 :     D005A-AC.ML
1106 :     D005B-FL.ML
1107 :     D005C-FL.ML
1108 :    
1109 :     M001A-AC.ML
1110 :     R002A-FL.ML
1111 :     R003A-FL.ML
1112 :     R011C-FL.ML
1113 :     R011D-FL.ML
1114 :     R011E-FL.ML
1115 :     R017J-FL.ML
1116 :     R017K-FL.ML
1117 :     R021b-FL.ML
1118 :     R031C-FL.ML
1119 :     R038A-AC.ML
1120 :     R100A-FL.ML
1121 :    
1122 :     S002A-FL.ML
1123 :     S002B-FL.ML
1124 :     S002C-FL.ML
1125 :     S002F-FL.ML
1126 :     Micro ML of Umea, Version 0.1, September 24, 1991
1127 :    
1128 :     Note that Micro ML of Umea does not support records. All programs
1129 :     listed below as failing because of the use of records are marked
1130 :     with an asterisk (*). When compiling the test programs, constructors
1131 :     were shown in another order than they appeared in the files. An
1132 :     example of a file, where the order was changed is R029A-AC.ML.
1133 :    
1134 :     The following programs were elaborated differently than expected:
1135 :    
1136 :     D001A-AC.ML
1137 :     D002A-AC.ML
1138 :     D004A-AC.ML
1139 :     D005A-AC.ML
1140 :     D005B-FL.ML
1141 :     D005C-FL.ML
1142 :    
1143 :     M001A-AC.ML * M001B-FL.ML *
1144 :     M003A-FL.ML
1145 :    
1146 :     R011C-FL.ML
1147 :     R011D-FL.ML
1148 :     R011E-FL.ML
1149 :     R017D-AC.ML
1150 :     R017G-AC.ML R017H-FL.ML
1151 :     R017J-FL.ML
1152 :     R017K-FL.ML
1153 :     R019C-FL.ML
1154 :     R020B-FL.ML
1155 :     R024A-AC.ML
1156 :     R032A-AC.ML
1157 :     R034B-AC.ML R038A-AC.ML *
1158 :    
1159 :     S002F-FL.ML
1160 :     S002G-FL.ML
1161 :     The following programs were elaborated differently than expected
1162 :     causing cryptic error messages:
1163 :    
1164 :     D006C-FL.ML
1165 :     D006D-FL.ML
1166 :     D006E-FL.ML
1167 :    
1168 :     R002A-FL.ML
1169 :     R003A-FL.ML
1170 :     R005A-AC.ML *
1171 :     R005B-FL.ML * R005C-FL.ML *
1172 :     R008A-AC.ML *
1173 :     R010B-FL.ML *
1174 :     R031B-FL.ML
1175 :     R031C-FL.ML
1176 :     R041A-AC.ML *
1177 :     R043A-AC.ML *
1178 :     R043B-FL.ML * R044A-AC.ML *
1179 :     R044B-FL.ML *
1180 :     R045A-AC.ML *
1181 :     R045B-FL.ML *
1182 :     R046A-AC.ML *
1183 :     R046B-FL.ML *
1184 :     R048A-AC.ML *
1185 :     R049A-AC.ML * R049B-FL.ML *
1186 :     R052A-AC.ML *
1187 :     R100A-FL.ML
1188 :    
1189 :     S001B-FL.ML *
1190 :     S001C-FL.ML *
1191 :     S001D-FL.ML *
1192 :    
1193 :     Moscow SML by Sergei Romanenko, Version 0.04b, August 1994
1194 :    
1195 :     The following programs failed:
1196 :    
1197 :     R027B-AC.ML
1198 :    
1199 :     The following programs caused a cryptic error messages:
1200 :    
1201 :     R005C-FL.ML
1202 :     R011B-FL.ML
1203 :    
1204 :    
1205 :     Moscow SML by Sergei Romanenko, Version 1.00, August 1994
1206 :    
1207 :     The following program caused a cryptic error message:
1208 :    
1209 :     R011B-FL.ML
1210 :    
1211 :     Standard ML of New Jersey, Version 0.93, February 15, 1993
1212 :    
1213 :     The following program was not evaluated correctly:
1214 :    
1215 :     D001A-AC.ML
1216 :    
1217 :     In the following declaration
1218 :    
1219 :     local
1220 :     exception A;
1221 :     exception B = A;
1222 :     in
1223 :     val test5 = 1 = (case B of A => 1 | B => 2 | _ => 3);
1224 :     end;
1225 :    
1226 :     test5 was evaluated to false. The reason is that the compiler did not
1227 :     match the exception B with the exception A. This does not comply
1228 :     with rule 139 [page 54]. However, this rule is part of the dynamic
1229 :     semantics for the Core and it is not tested thoroughly in this test
1230 :     suite.
1231 :    
1232 :    
1233 :     Standard ML of New Jersey, Version 0.93, port no. 2 of May 1994,
1234 :     OS/2
1235 :    
1236 :     This compiler behaved exactly like the UNIX version of SML of New
1237 :     Jersey.
1238 :    

root@smlnj-gforge.cs.uchicago.edu
ViewVC Help
Powered by ViewVC 1.0.0