Operations on source trees

Author(s): Jose F. Morales, The Ciao Development Team.

This module defines predicates to operate on directory trees containing source code (source trees).

The possible operations include enumerating, copying, and cleaning files or selection of files based on filters. See source_filter/1 for a description of the available filters.

Details of the walk algorithm

The walk algorithm (see internal walk/3) traverses a filtered directory structure and invokes actions (see internal action_hook/3, walk_action/1) parameterized by walk events (see internal walk_event/1) before entering a directory (enter event), when a file is processed (file event), and once the files in a directory are processed (exit event).

Filters restrict the walk to selected parts of the file tree. Filters are decomposed into formulas of basic filters, where a basic filter is a property of a regular file or directory (e.g., based on its name or path).

A normalized filter (see internal norm_filter/2) is given by the tuple of formulas (WalkP, DirP, FileP), where each component represents the following conditions for the walk algorithm:

WalkP: walk into the directory
DirP: treat directory files
FileP: treat the file (a regular file or a directory not marked for walk)

Given a base directory, the walk algorithm will perform the following operation on each file F:

   if F is a directory and WalkP(F):
      if DirP(F):
     Action 'enter' on F
     call recursively for each file in F
     Action 'exit' on F
   else:
      if FileP(F):
     Action 'file' on F

Efficiency and memory usage

Evaluation of filter checks is optimized to minimize accesses to the filesystem.
This code depends on the regular expression library, which is currently not optimized for performance.
Since file names are encoded as atoms, this code may suffer from exhaustion of the atom table on large file trees.

Examples

Find cleanable files in the current directory (recursively):

   ?- current_file_find([cleanable(src)], '.', X).

Find distributable packages:

   ?- current_file_find([proj(distributable), srctype(package)], '.', X).

Usage and interface

Library usage:
:- use_module(library(source_tree)).
Exports:
- Predicates:
  current_file_find/3, copy_file_tree/4, copy_file_tree/5, clean_file_tree/2, copy_file_or_dir/2, remove_dir/1, remove_file_or_dir/1, delete_glob/2, remove_glob/2, match_def/3, get_file_srctype/2.
- Regular Types:
  source_filter/1, precomp_level/1, file_srctype/1.

Documentation on exports

PREDICATEcurrent_file_find/3

Usage:current_file_find(Filter,BaseDir,FileName)

Enumerates recursively all files FileName (absolute file name) in BaseDir directory files that match the corresponding Filter.

Call and exit should be compatible with:
(source_tree:source_filter/1)Filter is a source file filter
(basic_props:atm/1)BaseDir is an atom.
(basic_props:atm/1)FileName is an atom.
The following properties should hold at call time:
(term_typing:nonvar/1)Filter is currently a term which is not a free variable.
(term_typing:nonvar/1)BaseDir is currently a term which is not a free variable.
(term_typing:var/1)FileName is a free variable.

PREDICATEcopy_file_tree/4

Usage:copy_file_tree(Filter,SrcDir,DestDir,Perms)

Copy the file tree from SrcDir to DestDir with permissions Perms

PREDICATEcopy_file_tree/5

Usage:copy_file_tree(Filter,SrcDir,DestDir,Perms,Owner)

Copy the file tree from SrcDir to DestDir with permissions Perms and owner Owner

PREDICATEclean_file_tree/2

Usage:clean_file_tree(PrecompLevel,Dir)

Unwind the precompilation level at directory Dir, cleaning the contents recursively (see untainted and cleanable(PrecompLevel) source_filter/1).

PREDICATEcopy_file_or_dir/2

Usage:copy_file_or_dir(FileName,DestDir)

Copy FileName file into DestDir directory. The path for DestDir is created if it does not exists. If FileName is a directory, all its contents are copied recursively.

PREDICATEremove_dir/1

Usage:remove_dir(Dir)

Delete the directory Dir and all its contents recursively. Throws exception if file does not exist.

PREDICATEremove_file_or_dir/1

Usage:remove_file_or_dir(FileName)

Delete FileName. If FileName is a directory, all its contents are deleted recursively. Ignore errors if file does not exist.

PREDICATEdelete_glob/2

Usage:delete_glob(Dir,Pattern)

Delete each file in directory Dir (non-recursively) that matches the glob pattern Pattern

PREDICATEremove_glob/2

Usage:remove_glob(Dir,Pattern)

Like delete_glob/2, but uses remove_dir/1 for each matching directory in Dir.

REGTYPEsource_filter/1

A filter for source files. A filter is a basic filter, an extended filter (defined on terms on other filters), and a list of basic or extended filters.

Basic filters:

true

Enumerate all files (recursively).

nonrec

Do not recurse into directories (directories will be treated as files).

untainted

Enumerate all files, except those that are backups or hold repository metadata.

srctype(SrcTypes)

Enumerate source files of any of the specified SrcTypes, a source type or list of source types. A source type (file_srctype/1) is any of typesmodule, package, or include files (for user or included files).

proj(TreeProj)

Obtain a tree projection based on marks. Projections allow discarding whole subtrees or individual files. The valid projection names and their marks are:

compilable: for each directory, discard the subtree if it contains the NOCOMPILE file, or ignore the files whose name match with any of the patterns listed in the file NOCOMPILEFILES.
testable: same with NOTEST and NOTESTFILES (implies compilable).
distributable: same with NODISTRIBUTE and NODISTRIBUTEFILES.
installable: same with NOINSTALL and NOINSTALLFILES.

precomp(PrecompLevel)

Enumerate files for the specified precompilation level PrecompLevel (see precomp_level/1).

cleanable(PrecompLevel)

Enumerate the files resulting from compilation of source files that must be cleaned in order reduce the precompilation to PrecompLevel. E.g., PrecompLevel=src cleans all compiler output, PrecompLevel=noa cleans only platform dependant files.

Extended filters:

compilable_module

Enumerate Ciao modules that are suitable for automatic compilation during bundle build (in the untainted proj(compilable) projection).

testable_module

Enumerate Ciao modules that can be compiled and tested (in the untainted proj(compilable) and proj(testable) projection).

distributable_precomp(PrecompLevel)

Enumerate files that can be distributed (in the untainted proj(distributable) projection and for the given precompilation level PrecompLevel).

Additionally, excludes the build directory and some temporary files.

installable_precomp(PrecompLevel)

Enumerate (source) files that can be installed (in the untainted proj(installable) projection and for the given precompilation level PrecompLevel).

Additionally, excludes the build directory and some temporary files.

Usage:source_filter(Filter)

Filter is a source file filter

REGTYPEprecomp_level/1

The valid precompilation levels are:

src: source files only
noa: sources and portable objects
bin: sources, portable, and architecture dependant objects (except special third-party files like elisp objects and Java classes)
full: like bin, including special third-party files.

Usage:precomp_level(PrecompLevel)

PrecompLevel is a pre-compilation level

PREDICATEmatch_def/3

No further documentation available for this predicate.

REGTYPEfile_srctype/1

Source file types, defined as:

file_srctype(module).
file_srctype(package).
file_srctype(include).

Usage:file_srctype(T)

T is a source file type

PREDICATEget_file_srctype/2

This operation should be relatively fast, since we only need to read the first term in the file. Modules start with a :- module/3 (or :- module/2) or :- package/1 directive.

Usage:get_file_srctype(FileName,Type)

FileName is a file of type Type (module, package, or include for included or user sources). Fails if the file does not seem Ciao code

Call and exit should be compatible with:
(basic_props:atm/1)FileName is an atom.
(source_tree:file_srctype/1)Type is a source file type
The following properties should hold at call time:
(term_typing:nonvar/1)FileName is currently a term which is not a free variable.

Documentation on imports

This module has the following direct dependencies:

System library modules:
system, pathnames, lists, sort, aggregates, regexp_code, system_extra, bundle_paths, stream_utils, read.
Packages:
prelude, initial, condcomp, assertions, assertions/assertions_basic, fsyntax, hiord, regtypes, isomodes.