Home My Page Projects Code Snippets Project Openings SML/NJ
Summary Activity Forums Tracker Lists Tasks Docs Surveys News SCM Files

SCM Repository

[smlnj] Annotation of /sml/trunk/tests/README
ViewVC logotype

Annotation of /sml/trunk/tests/README

Parent Directory Parent Directory | Revision Log Revision Log


Revision 548 - (view) (download)

1 : dbm 548 SML/NJ Regression Testing
2 :     =========================
3 :    
4 :     This directoy contains regression test suites and shell scripts for
5 :     running them.
6 :    
7 :     There are 5 active test suites, found in the directories:
8 :    
9 :     coresml core SML tests (based on tests from Peter Sestoft)
10 :     typing type checking
11 :     modules modules
12 :     bugs bug tests
13 :     basis Basis tests (based on tests from Peter Sestoft) [Note: filesys]
14 :    
15 :     (the sixth directory, "printing", is not currently active).
16 :    
17 :     Each test suite directory contains two subdirectories
18 :    
19 :     tests source files for tests
20 :     outputs corresponding output files [Note: output variants]
21 :    
22 :     and a file
23 :    
24 :     openbugs
25 :    
26 :     that lists test files with known problems in the current version
27 :     of SML/NJ (i.e. known regression failures).
28 :    
29 :    
30 :     Scripts:
31 :    
32 :     bin/testml.sh - performs tests in a suite
33 :     bin/process.sh - generates report of results
34 :     bin/dotest.sh - combines testml.sh and process.sh
35 :     bin/doallseq.sh - performs all suites one after another
36 :     bin/doallpar.sh - performs all suites in parallel
37 :    
38 :     Normally one will execute one of dotest.sh, doallseq.sh. or
39 :     doallpar.sh. The scripts are designed not to interfere with one
40 :     another, so that test suites can be executed concurrently on a given
41 :     architecture, and also on multiple machines (or varying architectures)
42 :     sharing the file system containing this directory.
43 :    
44 :     For a particular architecture and operating system, arch-os,
45 :     bin/testml.sh produces a file <testdir>/LOG.arch-os and
46 :     a directory <testdir>/bad.arch-os. process.sh <testdir> then
47 :     produces a file <testdir>/RESULTS.arch-os.
48 :    
49 :     Before running a regression test suite, any old copies of these files
50 :     and directories should be removed (if not, the scripts will complain).
51 :    
52 :     --------------------------------------------------------------------
53 :     bin/dotest.sh <testdir> [-diff] [-sml <filename>] - run tests in <testdir>
54 :    
55 :     <testdir>
56 :     the name of the test suite to be run (coresml, typing, etc).
57 :    
58 :     -sml <filename>:
59 :     the sml command to be used (e.g. "/usr/local/sml/bin/sml").
60 :     If this option is not supplied, the value of shell variable SML
61 :     is used instead, and if that is not defined, the default command
62 :     name "/usr/local/sml/bin/sml" is used.
63 :    
64 :     -diff
65 :     this option causes the diff between the test output and
66 :     the corresponding reference output (in <testdir>/outputs) to be
67 :     used in the RESULT file, overriding the default behavior, which
68 :     is to include the old and new outputs in full.
69 :    
70 :     --------------------------------------------------------------------
71 :     bin/doallseq.sh [-sml <filename>] - run all suites sequentially
72 :     bin/doallpar.sh [-sml <filename>] - run all suites in parallel
73 :    
74 :     -sml <filename>:
75 :     the sml command to be used (e.g. "/usr/local/sml/bin/sml").
76 :     If this option is not supplied, the value of shell variable SML
77 :     is used instead, and if that is not defined, the default command
78 :     name "/usr/local/sml/bin/sml" is used.
79 :    
80 :     --------------------------------------------------------------------
81 :     Example: (with the testing directory as current working directory)
82 :    
83 :     rm -rf bugs/LOG.sparc-solaris bugs/bad.sparc-solaris bugs/RESULTS.sparc-solaris
84 :     (or just rm -rf bugs/*.sparc-solaris)
85 :    
86 :     bin/dotest.sh bugs -diff -sml /usr/local/sml/110/bin/sml
87 :    
88 :     Then review the bugs/RESULTS.sparc-solaris file to see if there are
89 :     any discrepancies, which could be unexpected failures or bug fixes.
90 :    
91 :    
92 :     --------------------------------------------------------------------
93 :     Notes
94 :     -----
95 :    
96 :     [filesys] The test basis/tests/filesys.sml is sensitive to file
97 :     permissions and may produce spurious failures ("EXN" instead of "OK")
98 :     if the tester is not the owner of the files in basis/tests.
99 :    
100 :     [output variants] There are actually two versions of the outputs directory
101 :     for each test suite. These are named
102 :    
103 :     outputs.pre-110.5
104 :     outputs.post-110.5
105 :    
106 :     The first of these should be used for testing versions before 110.5,
107 :     while the second should be used for versions >= 110.5. Before running
108 :     any of the test scripts, output should be defined as a symbolic link
109 :     to one of these two directories. E.g. if testing 110.0.6, in each
110 :     test suite directory do:
111 :    
112 :     rm outputs
113 :     ln -s outputs.pre-110.5 outputs
114 :    
115 :     The reason for the two versions of outputs is that printing changed with
116 :     110.5 and there is an extra blank line at the end of each output file.
117 :    
118 :     Authors: Dave MacQueen <dbm@research.bell-labs.com>,
119 :     Lal George <george@research.bell-labs.com>

root@smlnj-gforge.cs.uchicago.edu
ViewVC Help
Powered by ViewVC 1.0.0