Home My Page Projects Code Snippets Project Openings SML/NJ
Summary Activity Forums Tracker Lists Tasks Docs Surveys News SCM Files

SCM Repository

[smlnj] Annotation of /tests/trunk/README
ViewVC logotype

Annotation of /tests/trunk/README

Parent Directory Parent Directory | Revision Log Revision Log


Revision 3109 - (view) (download)

1 : dbm 548 SML/NJ Regression Testing
2 :     =========================
3 :    
4 :     This directoy contains regression test suites and shell scripts for
5 :     running them.
6 :    
7 :     There are 5 active test suites, found in the directories:
8 :    
9 :     coresml core SML tests (based on tests from Peter Sestoft)
10 :     typing type checking
11 :     modules modules
12 :     bugs bug tests
13 :     basis Basis tests (based on tests from Peter Sestoft) [Note: filesys]
14 :    
15 :     (the sixth directory, "printing", is not currently active).
16 :    
17 :     Each test suite directory contains two subdirectories
18 :    
19 :     tests source files for tests
20 :     outputs corresponding output files [Note: output variants]
21 :    
22 :     and a file
23 :    
24 :     openbugs
25 :    
26 :     that lists test files with known problems in the current version
27 : dbm 552 of SML/NJ (i.e. known regression failures). [Note!: basis/tests setup]
28 : dbm 548
29 :    
30 :     Scripts:
31 :    
32 :     bin/testml.sh - performs tests in a suite
33 :     bin/process.sh - generates report of results
34 :     bin/dotest.sh - combines testml.sh and process.sh
35 :     bin/doallseq.sh - performs all suites one after another
36 :     bin/doallpar.sh - performs all suites in parallel
37 :    
38 :     Normally one will execute one of dotest.sh, doallseq.sh. or
39 :     doallpar.sh. The scripts are designed not to interfere with one
40 :     another, so that test suites can be executed concurrently on a given
41 :     architecture, and also on multiple machines (or varying architectures)
42 :     sharing the file system containing this directory.
43 :    
44 :     For a particular architecture and operating system, arch-os,
45 :     bin/testml.sh produces a file <testdir>/LOG.arch-os and
46 :     a directory <testdir>/bad.arch-os. process.sh <testdir> then
47 :     produces a file <testdir>/RESULTS.arch-os.
48 :    
49 :     Before running a regression test suite, any old copies of these files
50 :     and directories should be removed (if not, the scripts will complain).
51 :    
52 :     --------------------------------------------------------------------
53 :     bin/dotest.sh <testdir> [-diff] [-sml <filename>] - run tests in <testdir>
54 :    
55 :     <testdir>
56 :     the name of the test suite to be run (coresml, typing, etc).
57 :    
58 :     -sml <filename>:
59 :     the sml command to be used (e.g. "/usr/local/sml/bin/sml").
60 :     If this option is not supplied, the value of shell variable SML
61 :     is used instead, and if that is not defined, the default command
62 :     name "/usr/local/sml/bin/sml" is used.
63 :    
64 :     -diff
65 :     this option causes the diff between the test output and
66 :     the corresponding reference output (in <testdir>/outputs) to be
67 :     used in the RESULT file, overriding the default behavior, which
68 :     is to include the old and new outputs in full.
69 :    
70 :     --------------------------------------------------------------------
71 :     bin/doallseq.sh [-sml <filename>] - run all suites sequentially
72 :     bin/doallpar.sh [-sml <filename>] - run all suites in parallel
73 :    
74 :     -sml <filename>:
75 :     the sml command to be used (e.g. "/usr/local/sml/bin/sml").
76 :     If this option is not supplied, the value of shell variable SML
77 :     is used instead, and if that is not defined, the default command
78 :     name "/usr/local/sml/bin/sml" is used.
79 :    
80 :     --------------------------------------------------------------------
81 :     Example: (with the testing directory as current working directory)
82 :    
83 :     rm -rf bugs/LOG.sparc-solaris bugs/bad.sparc-solaris bugs/RESULTS.sparc-solaris
84 :     (or just rm -rf bugs/*.sparc-solaris)
85 :    
86 :     bin/dotest.sh bugs -diff -sml /usr/local/sml/110/bin/sml
87 :    
88 :     Then review the bugs/RESULTS.sparc-solaris file to see if there are
89 :     any discrepancies, which could be unexpected failures or bug fixes.
90 :    
91 :    
92 :     --------------------------------------------------------------------
93 :     Notes
94 :     -----
95 :    
96 :     [filesys] The test basis/tests/filesys.sml is sensitive to file
97 :     permissions and may produce spurious failures ("EXN" instead of "OK")
98 :     if the tester is not the owner of the files in basis/tests.
99 :    
100 : dbm 552 [basis/tests setup] After you have created tests directory (say by checking
101 : dbm 3109 out from the Chicago gforge SVN repository), be sure to read the instructions
102 : dbm 552 in tests/basis/tests/README. This file explains how to set up some
103 :     symbolic and hard links needed by the filesys.sml test.
104 :    
105 : dbm 548 [output variants] There are actually two versions of the outputs directory
106 :     for each test suite. These are named
107 :    
108 :     outputs.pre-110.5
109 :     outputs.post-110.5
110 :    
111 :     The first of these should be used for testing versions before 110.5,
112 :     while the second should be used for versions >= 110.5. Before running
113 :     any of the test scripts, output should be defined as a symbolic link
114 :     to one of these two directories. E.g. if testing 110.0.6, in each
115 :     test suite directory do:
116 :    
117 :     rm outputs
118 :     ln -s outputs.pre-110.5 outputs
119 :    
120 :     The reason for the two versions of outputs is that printing changed with
121 :     110.5 and there is an extra blank line at the end of each output file.
122 :    
123 :     Authors: Dave MacQueen <dbm@research.bell-labs.com>,
124 :     Lal George <george@research.bell-labs.com>

root@smlnj-gforge.cs.uchicago.edu
ViewVC Help
Powered by ViewVC 1.0.0