This history of change document is intended to provide a synopsis of the notable changes that have been made between versions of Pipelines. It will detail only those changes which are significant, specifically; changes which may affect the functionality of existing, user-written pipelines.

Home

Download

     ●

SPECS stage command: the NEXTWORD operand will no longer insert a SPACE into the output record, before appending data; if that data is null.

     ●

A new DELAY stage command: which waits until a particular time of day or until a specified interval of time has elapsed before copying a primary input stream record to its primary output stream. Canned messages #64 and #65 have been added to support this new stage command.

     ●

SHELLEXECUTE stage command: was not capturing all of the executed process’ STDOUT and STDERR output data. This has been corrected. The stage command has also been redesigned; the new design includes the BATCH operand; which specifies that should an input record argument contain a syntax error or invalid data, or that the actual command fails to execute; SHELLEXECUTE will issue the appropriate error message, but it will continue to read, process and execute input record commands until it reaches end-of-file on its primary input stream. SHELLEXECUTE will only support the BATCH operand argument; all other arguments will be defined by each primary input stream record. Canned message #66 has been added to support this change in functionality.

     ●

CALLPIPE stage command: was not capturing all of the called pipeline’s primary and secondary output data. This has been corrected.

     ●

REVERSE stage command: has been removed.

You will now have to use the REVERSE operand of the SPECS stage command to reverse record data. SPECS allows you to reverse the data in a range of word, field and column output specifications, whereas the REVERSE stage was only able to reverse the contents of an entire record.

The following example shows how you might use the SPECS stage command to reverse the contents of columns 10 through 20 of an input record:

pipe ...

     | specs 10-20 reverse

     | ...

     ●

SHELLEXECUTE stage command: did not support the WAIT and NOWAIT keywords; these have now been added, but they will not actually be useable until Pipelines 1.7 is released. Additionally Pipelines 1.6 issues an incorrect message #63, it currently reads:

‘You cannot specify the WAIT keyword in combination with the CMD keyword.’

It has now been corrected, and reads:

‘You cannot specify the NOWAIT keyword in combination with the CMD keyword.’

Documentation file: Messages.htm (which was missing any reference to canned message #63) now in includes the correct message.

     ●

LOCATE and NLOCATE stage commands: support for the numeric comparison operators: <, <=, ==, !=, >= and >.

     ●

FILELIST stage command: support for the alternate stage command name: LISTFILE.

     ●

A new DEAL stage command; which writes primary input stream records to its connected output streams in either sequential order; starting with the primary output stream, or some other order specified on its secondary input stream.

     ●

This version of Pipelines is currently only available for the .NET 2.0 platform.

     ●

A Pipelines Visual Studio/VC++ Stage Command API mechanism (phase 1: VS8 - VC++ 9).

The API provides all the initialisation and runtime routines that support the current builtin stage set; comprising stage command parsing, runtime extraction and stream connection routines and a stage DLL project wizard which creates a fully functional skeletal Pipelines stage DLL.

     ●

In order to keep the Pipelines download package to a reasonable and manageable size and at the same time provide me with the flexibility to publish Pipelines features which may have already been developed but not yet documented, as and when they are deemed fit for use; the Pipelines documentation is now available online and no longer provided with the download package.

     ●

FILELIST stage command: the file date and time values have been changed to represent ‘local-time’; this brings the field’s in line with the DOS DIR and Windows file date and time property values.

     ●

The default regular expression mode was incorrectly defined as 1; this has been changed to a value of 2 (the standard Perl regular expression parsing mode).

     ●

A re-design and extension of the SNAKE stage command.

SNAKE now only delays those records which comprise a complete matrix, that is to say, if you specify the numrows operand; once the stage has read enough input records to build an output matrix of numcols by numrows; the output matrix is written to the primary output stream, the allocations for that matrix are released and the stage begins building the next matrix. This re-design improves the performance of the stage by a significant factor (for multiple matrices).

SNAKE now supports a PAD keyword and pad character operand which allows the specification of the character which is used to extend shorter input records for output alignment.

     ●

An increase in record throughput; the streamline of the StageManager i/o request mechanism provides much needed improvement in most pipeline constructions.

     ●

A MONITOR option keyword; which allows you to turn off stall detection.

     ●

LITERAL stage command: the addition of a PREFACE and APPEND keyword; which specify the order in which the command argument and primary input stream records are processed.

     ●

A new SHELLEXECUTE stage command; which allows you to specify a (system) shell command, or indeed any other executable process, either synchronously or asynchronously. This addition of the SHELL stage command allows you to construct pipelines which perform housekeeping; file and directory operations; for example; deleting, renaming and attribute modification.

     ●

Ill-placed and unnecessary registry definitions have been removed; this enables the RUNPIPE stage command to operate much more efficiently.

     ●

The correction of some broken hyperlinks in the Filein.htm document.

     ●

Pipelines version 1.5 is the last version of Pipelines to target the .NET 1.0 platform; all future versions of Pipelines will target .NET 2.0 as a minimum platform requirement.

     ●

BUFFER stage command: a REVERSE keyword; which specifies that the contents of the buffer are written to the primary output stream in reverse record order.

     ●

< (FILEIN) stage command: a REVERSE keyword; which specifies that an input file is read in reverse record order.

The REVERSE process uses a bespoke read process which traverses an input file in reverse order. To ensure that the < (FILEIN), < (FILEIN) REVERSE and IN stage commands all produce identical output records (so that multiple, multi-stream pipelines are consistent in their handling of the records) all three processes now split their input at New-Line characters only; individual and groups of Carriage-Return characters are discarded.

     ●

CONSOLE stage command: determines if its primary output stream is connected when the stage is dispatched; if true, once CONSOLE determines that its primary output stream has become disconnected, the stage ends.

     ●

Per-pipeline support for a range of regular expression flavours.

     ●

Implementation of a regular expression option for the following stage commands:

The PATTERN, ANYOF and charrange options remain, where appropriate; as they provide the fastest and most efficient method of searching for expressions which can be described by a range of characters or a simple pattern mask.

     ●

Per-stage command history of change details.

     ●

A new stage command: SNAKE; which builds multi-column matrices.

     ●

CHANGE stage command: was not writing unchanged records to its secondary output stream; this has been corrected.

     ●

JOIN stage command: no longer discards null input records; JOIN concatenates consecutive input records, up to, but not including a null input record.

     ●

Other miscellaneous changes and corrections.

     ●

NLOCATE stage command: selected and unselected records were not being written to the correct output streams; this has been corrected.

     ●

CALLPIPE and OUT stage commands have been extended: Pipelines now allows you to call sub-routine pipelines which route selected and unselected output records back to its caller; where OUT stream 0 corresponds to the primary output stream of the CALLPIPE stage of the calling pipeline, and OUT stream 1 corresponds to the secondary output stream of the CALLPIPE stage of the calling pipeline. This enhancement is significant; it makes possible the construction of filter pipelines that operate in a similar fashion to many of the builtin stage commands.

     ●

Other miscellaneous changes and corrections.

     ●

FILEIN stage command: changed to deny write access to open input files. This is complementary to the FILEOUT and FILEAPPEND stage commands; which deny read access to open output files.

     ●

Documentation files: Zone.htm and Casei.htm no longer specify example pipelines which use the LOCATE stage command. LOCATE does not use the pre-process functionality any more; it now provides full multi-word-field-column selection and a non-case-sensitive selection comparison.

     ●

SPECS stage command: the operands STRIP, alignment LEFT, CENTER and RIGHT, now only remove spaces (x’20’) and tabs (x’09’) before positioning the data in the output record.

     ●

This document: History of change.htm has been moved from the install package to the tenfiftytwo.co.uk/pipelines website.

     ●

Example pipelines: Machine snapshot.ppl and MachineDiff.ppl have been changed. They both now correctly write to and read from the users’ My Documents folder.

     ●

Multi-pipeline processing

     ●

Multi-stream connectivity

     ●

Pipeline stall-detection and dump mechanism

     ●

Priority dispatch control

     ●

Argument and place-holder substitution

     ●

The following bultin stage commands: