Annotation of /sml/trunk/tests/README

Parent Directory | Revision Log

---

Revision 552 - (view) (download)

1 :	dbm	548	SML/NJ Regression Testing
2 :			=========================
3 :
4 :			This directoy contains regression test suites and shell scripts for
5 :			running them.
6 :
7 :			There are 5 active test suites, found in the directories:
8 :
9 :			coresml core SML tests (based on tests from Peter Sestoft)
10 :			typing type checking
11 :			modules modules
12 :			bugs bug tests
13 :			basis Basis tests (based on tests from Peter Sestoft) [Note: filesys]
14 :
15 :			(the sixth directory, "printing", is not currently active).
16 :
17 :			Each test suite directory contains two subdirectories
18 :
19 :			tests source files for tests
20 :			outputs corresponding output files [Note: output variants]
21 :
22 :			and a file
23 :
24 :			openbugs
25 :
26 :			that lists test files with known problems in the current version
27 :	dbm	552	of SML/NJ (i.e. known regression failures). [Note!: basis/tests setup]
28 :	dbm	548
29 :
30 :			Scripts:
31 :
32 :			bin/testml.sh - performs tests in a suite
33 :			bin/process.sh - generates report of results
34 :			bin/dotest.sh - combines testml.sh and process.sh
35 :			bin/doallseq.sh - performs all suites one after another
36 :			bin/doallpar.sh - performs all suites in parallel
37 :
38 :			Normally one will execute one of dotest.sh, doallseq.sh. or
39 :			doallpar.sh. The scripts are designed not to interfere with one
40 :			another, so that test suites can be executed concurrently on a given
41 :			architecture, and also on multiple machines (or varying architectures)
42 :			sharing the file system containing this directory.
43 :
44 :			For a particular architecture and operating system, arch-os,
45 :			bin/testml.sh produces a file <testdir>/LOG.arch-os and
46 :			a directory <testdir>/bad.arch-os. process.sh <testdir> then
47 :			produces a file <testdir>/RESULTS.arch-os.
48 :
49 :			Before running a regression test suite, any old copies of these files
50 :			and directories should be removed (if not, the scripts will complain).
51 :
52 :			--------------------------------------------------------------------
53 :			bin/dotest.sh <testdir> [-diff] [-sml <filename>] - run tests in <testdir>
54 :
55 :			<testdir>
56 :			the name of the test suite to be run (coresml, typing, etc).
57 :
58 :			-sml <filename>:
59 :			the sml command to be used (e.g. "/usr/local/sml/bin/sml").
60 :			If this option is not supplied, the value of shell variable SML
61 :			is used instead, and if that is not defined, the default command
62 :			name "/usr/local/sml/bin/sml" is used.
63 :
64 :			-diff
65 :			this option causes the diff between the test output and
66 :			the corresponding reference output (in <testdir>/outputs) to be
67 :			used in the RESULT file, overriding the default behavior, which
68 :			is to include the old and new outputs in full.
69 :
70 :			--------------------------------------------------------------------
71 :			bin/doallseq.sh [-sml <filename>] - run all suites sequentially
72 :			bin/doallpar.sh [-sml <filename>] - run all suites in parallel
73 :
74 :			-sml <filename>:
75 :			the sml command to be used (e.g. "/usr/local/sml/bin/sml").
76 :			If this option is not supplied, the value of shell variable SML
77 :			is used instead, and if that is not defined, the default command
78 :			name "/usr/local/sml/bin/sml" is used.
79 :
80 :			--------------------------------------------------------------------
81 :			Example: (with the testing directory as current working directory)
82 :
83 :			rm -rf bugs/LOG.sparc-solaris bugs/bad.sparc-solaris bugs/RESULTS.sparc-solaris
84 :			(or just rm -rf bugs/*.sparc-solaris)
85 :
86 :			bin/dotest.sh bugs -diff -sml /usr/local/sml/110/bin/sml
87 :
88 :			Then review the bugs/RESULTS.sparc-solaris file to see if there are
89 :			any discrepancies, which could be unexpected failures or bug fixes.
90 :
91 :
92 :			--------------------------------------------------------------------
93 :			Notes
94 :			-----
95 :
96 :			[filesys] The test basis/tests/filesys.sml is sensitive to file
97 :			permissions and may produce spurious failures ("EXN" instead of "OK")
98 :			if the tester is not the owner of the files in basis/tests.
99 :
100 :	dbm	552	[basis/tests setup] After you have created tests directory (say by checking
101 :			out from the Yale CVS repository), be sure to read the instructions
102 :			in tests/basis/tests/README. This file explains how to set up some
103 :			symbolic and hard links needed by the filesys.sml test.
104 :
105 :	dbm	548	[output variants] There are actually two versions of the outputs directory
106 :			for each test suite. These are named
107 :
108 :			outputs.pre-110.5
109 :			outputs.post-110.5
110 :
111 :			The first of these should be used for testing versions before 110.5,
112 :			while the second should be used for versions >= 110.5. Before running
113 :			any of the test scripts, output should be defined as a symbolic link
114 :			to one of these two directories. E.g. if testing 110.0.6, in each
115 :			test suite directory do:
116 :
117 :			rm outputs
118 :			ln -s outputs.pre-110.5 outputs
119 :
120 :			The reason for the two versions of outputs is that printing changed with
121 :			110.5 and there is an extra blank line at the end of each output file.
122 :
123 :			Authors: Dave MacQueen <dbm@research.bell-labs.com>,
124 :			Lal George <george@research.bell-labs.com>

---

root@smlnj-gforge.cs.uchicago.edu	ViewVC Help
Powered by ViewVC 1.0.0