Mainframe sort merge
| Company type | Software tool |
|---|---|
| Industry | Mainframe software |
| Founded | 1960s |
| Products | Sorting, merging, duplicating, filtering, aggregating, reformatting, concatenating, reporting utilities |
The Sort/Merge utility is a mainframe software tool created to organize records within files, combine already sorted files, or selectively duplicate records. These utilities utilize conventional sorting algorithms, frequently enhanced by proprietary improvements.
In the early days of mainframes, limited main memory capacity meant sorting operations often dealt with extensive datasets. As a result, initial Sort/Merge programs focused on optimizing sorting performance on secondary storage media such as tape or disk. By 1968, the OS/360 Sort/Merge provided five distinct "sequence distribution techniques" customized for the hardware in use.
The term "alias" SORT has traditionally indicated the preferred sort program installed—whether IBM's Sort/Merge or third-party alternatives like SYNCSORT and CASORT. DFSORT is frequently recognized by its program identifier ICEMAN (component ICE), which evolved from the original OS/360 Sort/Merge program IERRCO00 (component IER), also linked to the SORT alias.
Virtual storage systems
Prior to the System/370, all IBM mainframe operating systems included built-in sort/merge utilities. With the emergence of virtual storage environments such as DOS/VS and OS/VS, IBM separated software offerings and introduced chargeable sort/merge products. For OS/VS, IBM released 5734-SM1 (OS Sort/Merge) and later 5740-SM1 (OS/VS Sort/Merge), which was eventually renamed Data Facility Sort (DFSORT).
In 1990, IBM incorporated the BLOCKSET merge algorithm into DFSORT as an advancement over the original OS/360 Sort/Merge. Interestingly, BLOCKSET was initially developed by an IBM Systems Engineer in 1963 and later rediscovered for implementation decades afterward.
Usage
Sort/Merge ranks among the most extensively utilized applications within mainframe settings, generally accounting for approximately twenty percent of total processing capacity.
Contemporary Sort/Merge utilities offer capabilities beyond basic sorting and merging: they can duplicate files, filter records, aggregate data, eliminate duplicates, reformat records, concatenate data, and produce reports. Many use cases exploit these sophisticated processing functions rather than mere sorting. The product accommodates numerous "user exits," which may be load modules or object decks; DFSORT dynamically links these as required. Working storage datasets (SORTWK01 through SORTWKnn) can be allocated on disk or tape; however, the BLOCKSET algorithm mandates disk-based working storage. Increasing the number of working storage datasets typically improves overall performance.
Competition
The sort/merge function is critical in mainframe environments, with several vendors offering compatible solutions for IBM mainframes and their z/OS, z/VM, and z/VSE operating systems. Prominent sort/merge products include:
- DFSORT by IBM
- SyncSort by Syncsort, Inc.
- CA-Sort by CA Technologies
Additionally, some providers supply versions compatible with platforms such as Unix, Linux, and Windows.
Migration
Sort/Merge utilities play a crucial role when transitioning mainframe workloads to platforms like Unix, Linux, or Windows. Products such as MFSORT from Micro Focus and AHLSORT replicate DFSORT functionality outside of mainframe environments, facilitating business continuity during platform migrations.
IBM OS/360 SORT
Prior to the introduction of virtual storage operating systems, the size of input data sets exceeded the capacity of main memory for complete sorting. To address this, OS/360 SORT implemented replacement selection alongside a focus on sequence distribution techniques aimed at optimizing secondary storage "sort work" (SORTWK) files. These strategies were designed to effectively handle sequences of records that were partially sorted.
OS/360 SORT provided five distinct distribution methods:
- For magnetic tape:
- Balanced (BALN): demanded 12,000 bytes of main memory and 2x+1 tape drives (where x represents the number of input tapes, up to a maximum of 15).
- Polyphase (POLY): required 12,000 bytes and utilized 3 intermediate tapes; this method supported only a single input reel.
- Oscillating (OSCL): needed 21,000 bytes and a maximum of either x+2 or 4 intermediate tapes (x being the input volumes, capped at 15).
- For direct access devices:
- Balanced (BALN): called for 13,000 bytes and between 3 and 6 disk work areas; the sorting capacity was influenced by available storage resources.
- Crisscross (CRCX): unsuitable for IBM 2311 or IBM 2301 devices; this technique required 24,000 bytes and from 6 to 17 auxiliary work areas; its capacity depended on available storage.
IBM OS/VS SORT
The tape-based distribution techniques from OS/360 SORT continued to be used in the OS/VS SORT program under the designation "conventional techniques." Meanwhile, disk sorting methods were enhanced with four newly introduced techniques:
- FLR-Blockset designed for fixed-length records
- VLR-Blockset tailored for variable-length records
- Peerage applicable to fixed-length records
- Vale suitable for both fixed and variable-length records