Home My Page Projects Code Snippets Project Openings SML/NJ
Summary Activity Forums Tracker Lists Tasks Docs Surveys News SCM Files

SCM Repository

[smlnj] Diff of /sml/trunk/src/system/README
ViewVC logotype

Diff of /sml/trunk/src/system/README

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 537, Fri Feb 18 17:20:16 2000 UTC revision 578, Tue Mar 14 05:16:29 2000 UTC
# Line 1  Line 1 
1  Compiler Hacker's Guide to the new CM...  Compiler Hacker's Guide to the new CM...
2  ========================================  ========================================
3    
4  Last change: 2/2000  Last change: 3/2000
5    
6  * Libraries  * Libraries
7  -----------  -----------
# Line 19  Line 19 
19  the compiler ("*" means that I consider the library potentially useful  the compiler ("*" means that I consider the library potentially useful
20  in its own right):  in its own right):
21    
22          * basis.cm      - The SML'97 basis library    * basis.cm                            SML'97 Basis Library (pre-loaded)
23          - cm-hook.cm    - an internal library for organizational purposes    * smlnj-lib.cm                        SML/NJ Utility Library
24          - cm-lib.cm     - the library that implements CM's functionality    * html-lib.cm                         SML/NJ HTML Library
25          * comp-lib.cm   - a helper library for the compiler, MLRISC, and CM    * pp-lib.cm                           SML/NJ Pretty-print Library
26          * full-cm.cm    - the library that exports the public interface to  
27                            the compilation manager (i.e., structure CM)    * ml-yacc-lib.cm                      SML/NJ ML-Yacc runtime library
28          * minimal-cm.cm - exports a reduced version of structure CM;  
29                            this one is pre-registered at the top level    * smlnj/compiler/{alpha32,hppa,ppc,sparc,x86}.cm
30          * host-cmb.cm   - the library that exports the public interface to                                          cross-compiler libraries, exporting
31                            the bootstrap compiler (i.e., structure CMB)                                          structure <Arch>Compiler
32          - host-compiler-0.cm    * smlnj/compiler/current.cm           structure Compiler (current arch)
33                          - an internal library for organizational purposes    * smlnj/compiler/all.cm               all cross-compilers and all cross-CMBs
34          * host-compiler.cm  
35                          - the library that exports the public interface to    * smlnj/cm/minimal.cm                 minimal CM (pre-loaded)
36                            the visible compiler (i.e., structure Compiler)    * smlnj/cm/full.cm                    full structure CM (see manual)
37          - intsys.cm     - an internal library for organizational purposes    * smlnj/cm/tools.cm                   CM tools library
38                            (In fact, its the "root" of the main hierarchy.)  
39          * ml-yacc-lib.cm - needs no further comment    * smlnj/cmb/{alpha32,hppa,ppc,sparc,x86}-unix.cm
40          * smlnj-lib.cm  - needs no further comment                                          cross-bootstrap-compilers for Unix
41          * <arch>-compiler.cm - exports the Compiler structure for the                                          (structure <Arch>UnixCMB)
42                             given architecture (<Arch>Compiler).    * smlnj/cmb/ppc-macos.cm              ...for Mac (structure PPCMacosCMB)
43          * <arch>-<os>.cm - exports <Arch><OS>CMB for the given architecture-    * smlnj/cmb/x86-win32.cm              ...for Windoze (structure X86Win32CMB)
44                             OS combination.    * smlnj/cmb/current.cm                structure CMB (current arch/os)
45          * target-compilers.cm  
46                          - library exporting target-specific versions of    * smlnj/compiler.cm                   abbrev. for smlnj/compiler/current.cm
47                            structure Compiler and of structure CMB.    * smlnj/cm.cm                         abbrev. for smlnj/cm/full.cm
48                            This is a summary of all the <arch>-compiler.cm    * smlnj/cmb.cm                        abbrev. for smlnj/cmb/current.cm
49                            and <arch>-<os>.cm libraries above.  
50                            (The existence of these libraries is the moral    * comp-lib.cm                         Utility library for compiler
51                             equivalent of "CMB.retarget" in the old CM.)  
52          * viscomp-core.cm - library that implements the machine-independent    - smlnj/viscomp/core.cm               Compiler core functionality
53                            core of the compiler.    - smlnj/viscomp/{alpha32,hppa,ppc,sparc,x86}.cm
54          * viscomp-<arch>.cm - library that implements the visible compiler                                          Machine-specific parts of compiler
55                             for a given architecture.  
56      - smlnj/internal/{intsys,cm-lib,cm-hook,host-compiler-0}.cm
57                                            Glue that holds the interactive system
58                                            together
59    
60      * MLRISC/{MLRISC,Control,Lib,ALPHA,HPPA,PPC,SPARC,IA32}.cm
61                                            Various MLRISC bits
62    
63      * {mlyacc,mllex,mlburg}-tool.cm       CM plug-in libraries for common tools
64      * {grm,lex,burg}-ext.cm               CM plug-in libraries for common file
65                                            extensions
66    
67  * Before you can use the bootstrap compiler (CMB)...  * Before you can use the bootstrap compiler (CMB)...
68  ----------------------------------------------------  ----------------------------------------------------
69    
70  To be able to use CMB at all, you must first say  To be able to use CMB at all, you must first say
71    
72          CM.autoload "host-cmb.cm";          CM.autoload "smlnj/cmb.cm";
73    
74  after you start sml.  after you start sml.  Alternatively -- and perhaps more conveniently --
75    you can provide "smlnj/cmb.cm" as a command-line argument to sml:
76    
77  * Compiling the compiler -- a two-step procedure        $ sml smlnj/cmb.cm
 ------------------------------------------------  
78    
79  Until now (with the old CM), once we managed to run CMB.make() to  * Compiling the compiler
80  completion we had a directory full of binfiles that were ready to be  ------------------------
81  used by the boot procedure.  This is no longer the case.  
82    We are now back to the old scheme where a call to CMB.make() suffices to
83  The boot procedure now wants to use stable libraries (except for the  build a bootable set of files (libraries in our case).  CMB.make maintains
84  part that makes up the pervasive environment).  Having stable  two parallel hierarchies of derived files:
85  libraries around during development of these very libraries would be a  
86  bit annoying because if CM sees a stable library it will no longer    1. the binfile hierarchy ("binfiles"), containing compiled objects for
87  bother to check the corresponding source files -- even if they have       each individual ML source file; this hierarchy is rooted at
88  changed.  Therefore, libraries are not stabilized until you think you        <prefix>.bin.<arch>-<opsys>
89  are ready for that.  Thus, you should run:    2. the stable library hierarchy ("boot files"), containing library files
90         for each library that participates in building SML/NJ; this hierarchy
91          CMB.make ();       is rooted at
92          <prefix>.boot.<arch>-<opsys>
93  until you no longer get compile errors.  CMB.make will return true in  
94  this case.  Then you say:  The default for <prefix> is "sml".  It can be changed by using
95    CMB.make' with the new <prefix> as the optional string argument.
96          CMB.deliver ();  
97    CMB.make uses bootfiles after it has verified that they are consistent
98  This command creates a second directory parallel to the "bin"  with their corresponding binfiles.  Bootfiles do not need to be
99  directory -- the "boot" directory.  It will hold everything necessary  deleted in order for CMB.make to work correctly.
100  to bootstrap a new heap image.  You will probably find that  
101  CMB.deliver() compiles a number of additional files even though  To bootstrap a new system (using the runtime system boot loader), the
102  CMB.make completed successfully.  This is because CMB.make compiles  bootfiles _must_ be present, the binfiles need not be present (but
103  just those modules that will actually go into the heap image, but  their presence does not hurt either).
 CMB.deliver must also build the remaining files -- files that are part  
 of libraries to be stabilized but which are not used by the compiler.  
104    
105  You can reduce the number of extra files compiled and stabilized  You can reduce the number of extra files compiled and stabilized
106  during CMB.deliver at the expense of not building any cross-compilers.  during CMB.make at the expense of not building any cross-compilers.
107  For that, say  For that, say
108          #set (CMB.symval "LIGHT") (SOME 1);          #set (CMB.symval "LIGHT") (SOME 1);
109  before running CMB.deliver.  before running CMB.make.
   
 After you have made the boot directory, if you want to continue  
 developing the compiler (i.e., make changes to some sources,  
 recompile, etc.), you must first get rid of that boot directory.  
 Running the "makeml" script (see below) will automatically remove the  
 boot directory.  
   
 The names of "bin" and "boot" directories are  
   
         <prefix>.bin.<arch>-<os>  
   
 and  
   
         <prefix>.boot.<arch>-<os>  
   
 respectively, with "sml" being the default for <prefix>.  To change  
 the prefix, use CMB.make' and CMB.deliver' with the new prefix  
 provided as the optional string argument to these functions.  
110    
111  * Making the heap image  * Making the heap image
112  -----------------------  -----------------------
# Line 129  Line 119 
119  The "feel" of using makeml should be mostly as it used to.  However,  The "feel" of using makeml should be mostly as it used to.  However,
120  internally, there are some changes that you should be aware of:  internally, there are some changes that you should be aware of:
121    
122  1. The script will make a heap image and also move its associated  1. The script will make a heap image and build a separate library directory
123     libraries into a separate directory.     that contains links to the library files in the bootfile directory.
124    
125  2. There is no "-full" option anymore.  This functionality should  2. There is no "-full" option anymore.  This functionality should
126     eventually be provided by a library with a sufficiently rich export     eventually be provided by a library with a sufficiently rich export
# Line 142  Line 132 
132     option to actually make the image.  The argument to "-rebuild"     option to actually make the image.  The argument to "-rebuild"
133     is the <prefix> for the new bin and boot directories (see above).     is the <prefix> for the new bin and boot directories (see above).
134    
135  4. Unless you use "-rebuild", makeml will delete the boot directory  Makeml will not destroy the bootfile directory.
    (thus readying you for further "CMB.make();" runs).  
136    
137  * Testing a newly generated heap image  * Testing a newly generated heap image
138  --------------------------------------  --------------------------------------
# Line 151  Line 140 
140  If you use a new heap image by saying "sml @SMLload=..." then things  If you use a new heap image by saying "sml @SMLload=..." then things
141  will not go as you may expect because along with the new heap image  will not go as you may expect because along with the new heap image
142  should go those new stable libraries, but unless you do something  should go those new stable libraries, but unless you do something
143  about it, the new CM will look for its stable libraries in places  about it, the newly booted system will look for its stable libraries
144  where you stored your _old_ stable libraries.  in places where you stored your _old_ stable libraries.
145    
146  After you have made the new heap image, the new libraries are in a  After you have made the new heap image, the new libraries are in a
147  separate directory whose name is derived from the name of the heap  separate directory whose name is derived from the name of the heap
148  image.  The "testml" script that you also find here will run the heap  image.  (Actually, only the directory hierachy is separate, the
149  image and instruct it to look for its libraries in that new library  library files themselves are hard links.)  The "testml" script that you
150  directory.  also find here will run the heap image and instruct it to look for its
151    libraries in that new library directory.
152    
153  "testml" takes the <prefix> of the heap image as its first  "testml" takes the <prefix> of the heap image as its first
154  argument. All other arguments are passed verbatim to the ML process.  argument. All other arguments are passed verbatim to the ML process.
# Line 182  Line 172 
172  "installml" patches the ../../lib/pathconfig file to reflect any  "installml" patches the ../../lib/pathconfig file to reflect any
173  changes or additions to the path name mapping.  changes or additions to the path name mapping.
174    
175  Thus, after a successful CMB.deliver, you should say  Thus, after a successful CMB.make, you should say
176    
177          ./makeml          ./makeml
178    
# Line 199  Line 189 
189  * Cross-compiling  * Cross-compiling
190  -----------------  -----------------
191    
192  All cross-compilers live in the "target-compilers.cm" library.  You  All cross-compilers live in the "smlnj/compiler/all.cm" library.  You
193  must first say  must first say
194    
195          CM.autoload "target-compilers.cm";          CM.autoload "smlnj/compiler/all.cm";
196    
197  before you can access them.  (This step corresponds to the old  before you can access them.  (This step corresponds to the old
198  CMB.retarget call.)  After that, _all_ cross-compilers are available  CMB.retarget call.)  After that, _all_ cross-compilers are available
# Line 229  Line 219 
219  implementation of the basis library for the MacOS.)  implementation of the basis library for the MacOS.)
220    
221  Alternatively, you can select just the one single structure that you  Alternatively, you can select just the one single structure that you
222  are interested in by auto-loading <arch>-compiler.cm or <arch>-<os>.cm.  are interested in by auto-loading smlnj/compiler/<arch>.cm or
223    smlnj/cmb/<arch>-<os>.cm.
224  <arch> currently ranges over "alpha32", "hppa", "ppc", "sparc", and "x86.  <arch> currently ranges over "alpha32", "hppa", "ppc", "sparc", and "x86.
225  <os> can be either "unix" or "macos" or "win32".  <os> can be either "unix" or "macos" or "win32".
226  (Obviously, not all combinations are valid.)  (Obviously, not all combinations are valid.)
227    
228    Again, as with smlnj/cmb.cm, you can specify the .cm file as an
229    argument to the sml command:
230    
231        $ sml smlnj/compiler/all.cm
232    
233    or
234    
235        $ sml smlnj/cmb/alpha32-unix.cm
236    
237  * Path configuration  * Path configuration
238  --------------------  --------------------
239    
# Line 338  Line 338 
338    
339    mv <image>.<arch>-<osvariant> ../../bin/.heap/sml.<arch>-<osvariant>    mv <image>.<arch>-<osvariant> ../../bin/.heap/sml.<arch>-<osvariant>
340    
341  and then:  After this you must also move all libraries from <image>.libs/* to their
342    corresponding position in ../../lib.
343    
344    rm -r ../../lib/*.cm  Since this is cumbersome to do by hand, there is a script called
345    mv <image>.libs/*.cm ../../lib  "installml" that automates this task.  Using the script has the added
346    advantage that it will not clobber libraries that belong to other than
347  For convenience, there is a script called "installml" that automates  the current architecture.  (A rather heavy-handed "rm/mv" approach
348  this task.  Using the script has the added advantage that it will not  will delete all stable libraries for all architectures.)
349  clobber libraries that belong to other than the current architecture.  "installml" also patches the ../../lib/pathconfig file as necessary.
 (The rather heavy-handed "rm/mv" approach above will delete all stable  
 libraries for all architectures.)  "installml" also patches the  
 ../../lib/pathconfig file as necessary.  
350    
351  Of course, you can organize things differently for yourself -- the  Of course, you can organize things differently for yourself -- the
352  path configuration mechanism should be sufficiently flexible.  path configuration mechanism should be sufficiently flexible.
# Line 384  Line 382 
382  library, n for n sub-groups), there will be just one file representing  library, n for n sub-groups), there will be just one file representing
383  the one stable archive (per architecture/os) for the whole thing.  For  the one stable archive (per architecture/os) for the whole thing.  For
384  example, I structured the standard basis into one library with two  example, I structured the standard basis into one library with two
385  sub-groups, but once you compile it (CMB.deliver) there is only one  sub-groups, but once you compile it (CMB.make) there is only one
386  stable file that represents the whole basis library.  If groups were  stable file that represents the whole basis library.  If groups were
387  allowed to appear in more than one library, then stabilization would  allowed to appear in more than one library, then stabilization would
388  duplicate the group (its code, its environment data structures, and  duplicate the group (its code, its environment data structures, and
# Line 399  Line 397 
397  * Pervasive environment, core environment, the init group "init.cmi"  * Pervasive environment, core environment, the init group "init.cmi"
398  -------------------------------------------------------------------------  -------------------------------------------------------------------------
399    
400  CMB.make (or CMB.deliver) starts out by building and compiling the  CMB.make starts out by building and compiling the
401  "init group".  This group cannot be described in the "usual" way  "init group".  This group cannot be described in the "usual" way
402  because it uses "magic" in three ways:  because it uses "magic" in three ways:
403   - it is used to later tie in the runtime system   - it is used to later tie in the runtime system
404   - it builds the "core" environment   - it exports the "core" environment
405   - it builds the "pervasive" environment   - it exports the "pervasive" environment
406    
407  The pervasive environment no longer includes the entire basis library  The pervasive environment no longer includes the entire basis library
408  but only non-modular bindings (top-level bindings of variables and  but only non-modular bindings (top-level bindings of variables and
409  types).  types).
410    
411  CM cannot automatically determine dependencies for the init group  CM cannot automatically determine dependencies (or exports) for the
412  source files, but it still does use its regular cutoff recompilation  init group source files, but it still does use its regular cutoff
413  mechanism.  Therefore, dependencies must be given explicitly.  This is  recompilation mechanism.  Therefore, dependencies must be given
414  done by a special description file which currently lives in  explicitly.  This is done by a special description file which
415  Init/init.cmi.  See the long comment at the beginning of that file for  currently lives in ../system/smlnj/init/init.cmi.  See the long comment
416  more details.  at the beginning of that file for more details.
417    
418  After it is built, init.cmi can be used as an "ordinary" library by  After it is built, smlnj/init/init.cmi can be used as an "ordinary" library
419  other libraries.  (This is done, for example, by the implementation of  by other libraries.  (This is done, for example, by the implementation of
420  the Basis library.)  Access to "init.cmi" is protected by the  the Basis library.)  Access to "smlnj/init/init.cmi" is protected by the
421  privilege named "primitive".  privilege named "primitive".  Also, note that the .cmi-file is not
422    automatically recognized as as CM description file.  Therefore, it
423    must be given an explicit member class:
424    
425         smlnj/init/init.cmi : cm
426    
427  * Autoloader  * Autoloader
428  ------------  ------------
429    
430  The new system heavily relies on the autoloader.  As a result, almost  The new system heavily relies on the autoloader.  As a result, almost
431  no static environments need to get unpickled at bootstap time.  The  no static environments need to get unpickled at bootstrap time.  The
432  construction of such environments is deferred until they become  construction of such environments is deferred until they become
433  necessary.  Because of this, I was able to reduce the size of the heap  necessary.  Thanks of this, it was possible to reduce the size of the
434  image by more than one megabyte (depending on the architecture).  The  heap image by more than one megabyte (depending on the architecture).
435  downside (although not really terribly bad) is that there is a short  The downside (although not really terribly bad) is that there is a
436  wait when you first touch an identifier that hasn't been touched  short wait when you first touch an identifier that hasn't been touched
437  before.  (I acknowledge that the notion of "short" may depend on your  before.  (I acknowledge that the notion of "short" may depend on your
438  sense of urgency. :-)  sense of urgency. :-)
439    
# Line 448  Line 450 
450  which can easily be made accessible via CM.autoload (or, non-lazily,  which can easily be made accessible via CM.autoload (or, non-lazily,
451  via CM.make):  via CM.make):
452    
453          full-cm.cm              - provides the actual ("full") structure CM          smlnj/cm.cm             - provides the actual ("full") structure CM
454                                    as described in the CM manual                                    as described in the CM manual
455          host-compiler.cm        - provides "structure Compiler"          smlnj/cm/full.cm        - same as smlnj/cm.cm
456          host-cmb.cm             - provides "structure CMB"          smlnj/compiler.cm       - provides "structure Compiler"
457          target-compilers.cm     - provides "structure <Arch>Compiler" and          smlnj/compiler/current.cm - same as smlnj/compiler.cm
458            smlnj/cmb.cm            - provides "structure CMB"
459            smlnj/cmb/current.cm    - same as smlnj/cmb.cm
460            smlnj/compiler/all.cm   - provides "structure <Arch>Compiler" and
461                                    "structure <Arch><OS>CMB" for various                                    "structure <Arch><OS>CMB" for various
462                                    values of <Arch> and <OS>                                    values of <Arch> and <OS>
463          smlnj-lib.cm            - the SML/NJ library          smlnj-lib.cm            - the SML/NJ library
# Line 559  Line 564 
564  without having to move large source trees at the same time. (See my  without having to move large source trees at the same time. (See my
565  new build/install.sh script for examples of that.)  new build/install.sh script for examples of that.)
566    
567    It is possible to simulate aliases (in a way that is safer than the
568    original alias mechanism).  For example, the root.cm file (which is the
569    root of the whole system as far as CMB.make is concerned) acts as an
570    alias for smlnj/internal/intsys.cm.  In this case, root.cm is a group
571    to avoid having a (trivial) stable library file built for it.
572    
573    A library can act as an "alias" for another library if it has a
574    verbatim copy of the export list and mentions the other library as its
575    only member.  Examples for this are smlnj/cm.cm (for smlnj/cm/full.cm),
576    smlnj/compiler.cm (for smlnj/compiler/current.cm), etc.
577    
578  * Don't use relative or absolute pathnames to refer to libraries  * Don't use relative or absolute pathnames to refer to libraries
579  ----------------------------------------------------------------  ----------------------------------------------------------------
580    
581  Don't use relative or absolute pathnames to refer to libraries.  If  Don't use relative or absolute pathnames to refer to libraries.  If
582  you do it anyway, you'll get an appropriate warning at the time when  you do it anyway, you'll get an appropriate warning at the time when
583  you do CMB.deliver().  If you use relative or absolute pathnames to  you do CMB.make().  If you use relative or absolute pathnames to
584  refer to library B from library A, you will be committed to keeping B  refer to library B from library A, you will be committed to keeping B
585  in the same relative (to A) or absolute location.  This, clearly,  in the same relative (to A) or absolute location.  This, clearly,
586  would be undesirable.  would be undesirable.

Legend:
Removed from v.537  
changed lines
  Added in v.578

root@smlnj-gforge.cs.uchicago.edu
ViewVC Help
Powered by ViewVC 1.0.0