|
Pipelines by TenFiftyTwo |
Pipelines is inspired by and based
upon CMS Pipelines; an enterprise systems utility originally designed and
developed by John Hartmann of IBM. Pipelines allows you to modify the contents
of a text/data file or files, quickly and easily. You can specify that only
certain sections of a file are to be changed; you can confine those changes to
a column, word or field range, translate words and phrases, discard or insert
new lines of data. You can perform a whole range of operations on a file or
files, using only a simple set of commands.
Pipelines build’s on the concept of
directing the output of one process to the input of another, commonly known as
pipelining. It is an old idea and almost all operating systems support an
implementation of varying degree of usefulness. In general they support the
linear, single-stream model; where if you lay each process out in a straight
line, data starts in the first process, passing into the next where it is changed
in some way, and so on down the pipeline chain in a sequential fashion until it
reaches a sink. For example:
stage1 | stage2
| stage3 | ... | stagen
Pipelines builds on this mechanism;
allowing you to create multi-stream pipelines, where the topology is no longer
horizontal and linear, but two-dimensional; where the records travel up and
down the pipeline chain through intersecting joints which control the flow of
data. Multi-stream pipelines allow you to select and operate on specific sets
of records; routing unselected records through a joint into and out of other
sections of the pipeline.
Pipelines treats its input data as
lines or records, reading them one at
a time from its input and writing them one at a time to its output. As such and
unless the entire input needs to be loaded into memory-storage; Pipelines only
consumes a fraction of the memory that might otherwise be required, as only a
handful of records are ever in the
pipeline at any one time.
Pipelines allows you to operate on
files in a single pass; isolating sections of the file without having to
needlessly buffer or sort the data simply in order to maintain the relative
record order. Consider the following simple pipeline, which, utilising only 6 stages, reads the file: myfile.txt and in a single-pass, changes
the word hello to goodbye only in records that contain the
word friend.
pipe (endchar ?) < myfile.txt | a: locate ‘friend’ | change ‘hello’ ‘goodbye’ | b: faninany | > myfile.txt ? a: | take * | b:
Pipelines comprises a range of
input, output, selection and transformation stages
which cover a spectrum of manipulation functions; splitting records, stripping
characters, joining records, collating and sorting and more. On the whole,
similar operations are performed by a single stage; which means that you do not have to remember the names of an
unnecessarily lengthy list of stages.
For example; stripping characters from a record, Pipelines provides a single stage called STRIP which removes
characters from the beginning and/or the end of a record.
With Pipelines, the pipeline can be
specified on the system command-line, in a batch file or in a Pipelines file,
ext (.PPL). You design the pipeline
in your favourite editor and save it; to execute the pipeline you simply double
click the file icon and Pipelines will launch it. You can specify pipelines
which accept arguments which substitute stage
operands and even stage names and coupled with the capability to
connect pipelines together, this allows you build a range of utility pipelines
that can be called upon whenever you need them.
Pipelines is general purpose; it
has not been developed with any particular field in mind, it is simply a line/record
orientated textual processing utility that is useful for manipulating data. The
design of Pipelines is essentially a compromise between speed and flexibility.
A bespoke, dedicated program may out-perform Pipelines, However, with a
dedicated program; each time your requirements change that means altering the
source code (if it is compiled; that means re-building it as well). This is not
a problem when the program is small or simplistic. But, when we start to talk about
pattern, field, word and column selection, recursive sorting, collating,
splitting and joining records from multiple input files, possibly large files,
then we have a different scenario. Pipelines is designed with this type of
processing in mind; it is intended to offer a quick and efficient processing
utility that can help you manipulate data into a format that suits your needs.
You may find Pipelines of use in
cases where you might otherwise have to write a program to solve the problem
and it may well save you some time and effort that could be better spent on
other tasks.
Pipelines is free; there are no
evaluation caveats, you may download it and use it as you please.
|
|