CMS Pipelines
| Company type | Software functionality |
|---|---|
| Industry | Computer software |
| Founded | 1980 |
| Founder | John Hartmann |
| Products | Pipeline processing for VM/CMS |
| Website | www |
CMS Pipelines is a functionality within the VM/CMS operating system that allows users to construct and execute pipelines. In these pipelines, programs sequentially process streams of records, each program receiving input and delivering output via a device-agnostic interface, which facilitates versatile program combinations.
Overview
The CMS command PIPE is provided by CMS Pipelines and takes a pipeline description as its parameter. PIPE orchestrates the selection and linking of programs to handle data streams.
Because CMS programs do not inherently support device-independent stdin and stdout, CMS Pipelines offers a built-in collection of utility programs designed for pipeline integration. These built-in utilities interface directly with the operating system to perform various tasks.
On CMS, data is organized as logical records rather than continuous byte streams; for textual data, each line corresponds to one record. Data passed between pipeline stages consists of these logical records.
Users may execute pipeline commands interactively or embed them within EXEC scripts, combining standard built-in utilities with user-created REXX programs.
Example
The following example demonstrates reading a disk file, separating records that contain "Hello" from those that do not, appending "World!" to the matching records, converting non-matching records to uppercase, and then merging all results into a new file:
PIPE (end ?)
< input txt
| a: locate /Hello/
| insert / World!/ after
| i: faninany
| > newfile txt a
? a:
| xlate upper
| i:
In this pipeline, the < operator reads the input file. The locate command divides the stream: records matching "Hello" proceed to insert, which appends text and forwards output to faninany. Records without matches are routed to xlate, where they are converted to uppercase before also feeding into faninany. The combined output is then written to the new file.
The pipeline structure consists of two linked pipelines separated by the end marker (?). Records flow through either path without buffering delays, maintaining order when merged at faninany. For readability, the example is depicted vertically; in actual use, the pipeline command is entered as a single line.
Features
This straightforward pipeline model is enhanced with several advanced features:
- Programs can define sub-pipelines to process entire or partial input streams.
- Intersecting pipeline networks enable programs to participate in multiple pipelines concurrently, handling several data streams simultaneously.
- Data exchanged between stages consists of structured records, allowing operations on individual records without needing buffers for line delimiters.
- Typically, stages access input in locate mode and produce output before consuming input, eliminating data copying and supporting predictable multi-stream record processing.
- Pipelines support dynamic modification: programs can replace themselves, insert segments before or after their position, or generate new pipeline specifications from existing pipeline data.
CMS Pipelines also improves program reliability through mechanisms such as:
- Any syntax errors in the pipeline specification or within any program cause suppression of the entire pipeline execution.
- The CMS Pipelines dispatcher manages program initialization and resource allocation, ensuring that irreversible actions occur only after all programs have validated their arguments. It also oversees resource cleanup when programs terminate.
- Data flow errors are detectable by all participating programs, preventing unintended consequences such as accidental overwriting of files.
History
The inception of CMS Pipelines dates back to 1980 when John Hartmann at IBM Denmark initiated its development. Throughout the 1980s, IBM offered it as a standalone product before incorporating it into VM/ESA by the end of 1991. Subsequent VM updates included enhancements to the software until version 1.1.10, released with VM/ESA 2.3 in 1997, after which its functionality was effectively frozen. Since that time, the most recent release has been accessible for download via the CMS Pipelines website for users interested in experimenting with new features.
In November 2016, CMS Pipelines made a comeback with the launch of z/VM version 6.4.
The TSO variant was introduced in 1995 under the name BatchPipeWorks as part of the BatchPipes/MVS suite, and IBM Denmark managed this version until 2010. Both CMS and TSO editions originate from a unified source code and are collectively referred to as CMS/TSO Pipelines, with detailed documentation provided in the Author's Edition.
Additionally, open-source versions have been created in both Java and Swift.
See also
References