BatchPipes
| Company type | Software |
|---|---|
| Industry | Computer software |
| Founded | Late 1980s to early 1990s |
| Headquarters | Monaco |
| Products | Batch job handling tool for IBM mainframes |
| Parent | IBM |
On IBM mainframe systems, BatchPipes serves as a batch job handling tool operating under MVS/ESA and its successors—OS/390 and z/OS.
Core function
Sequential (QSAM and BSAM) data sets traditionally restrict simultaneous access by multiple jobs, resulting in a file-level interlock.
BatchPipes facilitates the "piping" of data between jobs, enabling them to run concurrently and often removing the necessity to store data temporarily on disk before rereading. This approach shortens the total elapsed time from when the producing job begins until the consuming job completes.
It implements a short queue holding records exchanged between writer and reader, allowing record-level interlock. Here, the writer appends records at the queue's rear while the reader processes from the front, supporting parallel operation.
A sort represents a unique scenario: all input records must be consumed before output generation can commence, preventing simultaneous input and output phases. Nevertheless, input processing can overlap with the preceding job’s output, and output can overlap with downstream jobs reading sorted results.
Advanced pipe topologies
More intricate arrangements beyond "one writer one reader" include:
- "Two readers one writer," which balances throughput by preventing a faster writer from being delayed by a slower reader.
- Jobs functioning simultaneously as reader and writer—such as those performing record transformations—can be incorporated using tools like IBM's DFSORT or BatchPipeWorks (a component of BatchPipes).
Criticism
A primary difficulty lies in scheduling reader and writer jobs to execute concurrently, which may prove impractical in batch processing environments. Furthermore, if any pipeline job fails, recovery becomes more complex than simply restarting individual jobs. These issues have limited BatchPipes’ adoption in certain settings.
BatchPipePlex
BatchPipes utilizes the IBM mainframe Coupling Facility to move data among members of an IBM Parallel Sysplex through the BatchPipePlex mechanism.
BatchPipeWorks
Inspired by IBM's CMS Pipelines on the VM/ESA platform, BatchPipes incorporates pipeline stages that enable additional processing within the pipeline itself without requiring extra batch jobs.
History
Developed during the late 1980s and early 1990s, BatchPipes Version 1 aimed to speed up MVS/ESA batch workflows. In 1997, its capabilities were merged into IBM’s SmartBatch product, which combined features from BMC’s DataAccelerator and BatchAccelerator. SmartBatch was discontinued in April 2000.
APT International, headquartered in Monaco, introduced a rival solution named WARP. Shortly thereafter, IBM rebranded its OS/2 operating system as OS/2 Warp 4, creating a marketing conflict with WARP that triggered seven years of litigation at the Tribunal de grande instance de Paris.
Subsequently, BatchPipes Version 2 was launched, integrating Version 1 with SmartBatch’s innovations such as BatchPipePlex and BatchPipeWorks. BatchPipes Version 2 continues to be offered by IBM.
See also
- Pipeline (Unix), the foundational idea
- CMS Pipelines
- ESA/390, a mainframe system architecture