Operations on source trees

Author(s): Jose F. Morales, The Ciao Development Team.

This module defines predicates to operate on directory trees containing source code (source trees).

The possible operations include enumerating, copying, and cleaning files or selection of files based on filters. See source_filter/1 for a description of the available filters.

Details of the walk algorithm

The walk algorithm (see internal walk/3) traverses a filtered directory structure and invokes actions (see internal action_hook/3, walk_action/1) parameterized by walk events (see internal walk_event/1) before entering a directory (enter event), when a file is processed (file event), and once the files in a directory are processed (exit event).

Filters restrict the walk to selected parts of the file tree. Filters are decomposed into formulas of basic filters, where a basic filter is a property of a regular file or directory (e.g., based on its name or path).

A normalized filter (see internal norm_filter/2) is given by the tuple of formulas (WalkP, DirP, FileP), where each component represents the following conditions for the walk algorithm:

WalkP
walk into the directory
DirP
treat directory files
FileP
treat the file (a regular file or a directory not marked for walk)

Given a base directory, the walk algorithm will perform the following operation on each file F:

   if F is a directory and WalkP(F):
      if DirP(F):
         Action 'enter' on F
         call recursively for each file in F
         Action 'exit' on F
   else:
      if FileP(F):
         Action 'file' on F

Efficiency and memory usage

  • Evaluation of filter checks is optimized to minimize accesses to the filesystem.
  • This code depends on the regular expression library, which is currently not optimized for performance.
  • Since file names are encoded as atoms, this code may suffer from exhaustion of the atom table on large file trees.

Examples

Find cleanable files in the current directory (recursively):

   ?- current_file_find([cleanable(src)], '.', X).
   

Find distributable packages:

   ?- current_file_find([proj(distributable), srctype(package)], '.', X).
   

Documentation on exports

Usage:current_file_find(Filter,BaseDir,FileName)

Enumerates recursively all files FileName (absolute file name) in BaseDir directory files that match the corresponding Filter.

Usage:copy_file_tree(Filter,SrcDir,DestDir,Perms)

Copy the file tree from SrcDir to DestDir with permissions Perms

    Usage:copy_file_tree(Filter,SrcDir,DestDir,Perms,Owner)

    Copy the file tree from SrcDir to DestDir with permissions Perms and owner Owner

      Usage:clean_file_tree(PrecompLevel,Dir)

      Unwind the precompilation level at directory Dir, cleaning the contents recursively (see untainted and cleanable(PrecompLevel) source_filter/1).

        Usage:copy_file_or_dir(FileName,DestDir)

        Copy FileName file into DestDir directory. The path for DestDir is created if it does not exists. If FileName is a directory, all its contents are copied recursively.

          PREDICATEremove_dir/1

          Usage:remove_dir(Dir)

          Delete the directory Dir and all its contents recursively. Throws exception if file does not exist.

            Usage:remove_file_or_dir(FileName)

            Delete FileName. If FileName is a directory, all its contents are deleted recursively. Ignore errors if file does not exist.

              PREDICATEdelete_glob/2

              Usage:delete_glob(Dir,Pattern)

              Delete each file in directory Dir (non-recursively) that matches the glob pattern Pattern

                PREDICATEremove_glob/2

                Usage:remove_glob(Dir,Pattern)

                Like delete_glob/2, but uses remove_dir/1 for each matching directory in Dir.

                  A filter for source files. A filter is a basic filter, an extended filter (defined on terms on other filters), and a list of basic or extended filters.

                  Basic filters:

                  true
                  Enumerate all files (recursively).

                  nonrec
                  Do not recurse into directories (directories will be treated as files).

                  untainted
                  Enumerate all files, except those that are backups or hold repository metadata.

                  srctype(SrcTypes)
                  Enumerate source files of any of the specified SrcTypes, a source type or list of source types. A source type (file_srctype/1) is any of typesmodule, package, or include files (for user or included files).

                  proj(TreeProj)
                  Obtain a tree projection based on marks. Projections allow discarding whole subtrees or individual files. The valid projection names and their marks are:

                  compilable
                  for each directory, discard the subtree if it contains the NOCOMPILE file, or ignore the files whose name match with any of the patterns listed in the file NOCOMPILEFILES.

                  testable
                  same with NOTEST and NOTESTFILES (implies compilable).

                  distributable
                  same with NODISTRIBUTE and NODISTRIBUTEFILES.

                  installable
                  same with NOINSTALL and NOINSTALLFILES.

                  precomp(PrecompLevel)
                  Enumerate files for the specified precompilation level PrecompLevel (see precomp_level/1).

                  cleanable(PrecompLevel)
                  Enumerate the files resulting from compilation of source files that must be cleaned in order reduce the precompilation to PrecompLevel. E.g., PrecompLevel=src cleans all compiler output, PrecompLevel=noa cleans only platform dependant files.

                  Extended filters:

                  compilable_module
                  Enumerate Ciao modules that are suitable for automatic compilation during bundle build (in the untainted proj(compilable) projection).

                  testable_module
                  Enumerate Ciao modules that can be compiled and tested (in the untainted proj(compilable) and proj(testable) projection).

                  distributable_precomp(PrecompLevel)
                  Enumerate files that can be distributed (in the untainted proj(distributable) projection and for the given precompilation level PrecompLevel).

                  Additionally, excludes the build directory and some temporary files.

                  installable_precomp(PrecompLevel)
                  Enumerate (source) files that can be installed (in the untainted proj(installable) projection and for the given precompilation level PrecompLevel).

                  Additionally, excludes the build directory and some temporary files.

                  Usage:source_filter(Filter)

                  Filter is a source file filter

                    The valid precompilation levels are:
                    src
                    source files only
                    noa
                    sources and portable objects
                    bin
                    sources, portable, and architecture dependant objects (except special third-party files like elisp objects and Java classes)
                    full
                    like bin, including special third-party files.

                    Usage:precomp_level(PrecompLevel)

                    PrecompLevel is a pre-compilation level

                      PREDICATEmatch_def/3
                      No further documentation available for this predicate.

                      Source file types, defined as:
                      file_srctype(module).
                      file_srctype(package).
                      file_srctype(include).
                      

                      Usage:file_srctype(T)

                      T is a source file type

                        This operation should be relatively fast, since we only need to read the first term in the file. Modules start with a :- module/3 (or :- module/2) or :- package/1 directive.

                        Usage:get_file_srctype(FileName,Type)

                        FileName is a file of type Type (module, package, or include for included or user sources). Fails if the file does not seem Ciao code

                        Documentation on imports

                        This module has the following direct dependencies:

                        Known bugs and planned improvements

                        • (bug) Does not work with CIAOCACHEDIR enabled
                        • (documentation) Add more examples
                        • (code) This code and documentation could be greatly improved by language extensions (like hooks definitions)
                        • (code) Use (;)/2 and (,)/2 for filters
                        • (code) Use dynamic facts instead of peep_p (the implementation may be simpler and more efficient)
                        • (code) Merge with bundle_scan.pl? (at least some predicates)
                        • (performance) Optimize regexp/glob matching
                        • (performance) We need atom GC
                        • (feature) Use a single PROPS or META file to store metadata?
                        • (feature) Extend and document all filter expressions
                        • (feature) Maybe WalkP can be inserted into DirP and FileP (see appendix)
                        • (feature) Add filters for system:file_property
                        • (feature) OR-composition of filters is not supported (we need a more involved algorithm that avoid duplicates)
                        • (feature) Add a predoc filter? (similar to precomp)
                        • Implement a copy that preserves original modes, perms, and owner? (like cp -p)