Vowpal Wabbit
| Company type | Open-source software |
|---|---|
| Industry | Machine learning software |
| Founder | Yahoo! Research |
Area served | Worldwide |
Key people | John Langford |
| Products | Machine learning platform |
| Owner | Microsoft Research |
| Parent | Microsoft Research |
| Website | github |
Vowpal Wabbit (abbreviated as VW) is a high-performance open-source machine learning platform designed for online and interactive learning. Initially created at Yahoo! Research, it is currently under the stewardship of Microsoft Research. Led by John Langford, VW is well-known for its support of interactive learning paradigms such as Contextual Bandits, Active Learning, and directed Reinforcement Learning. The system provides a scalable and efficient out-of-core framework that accommodates numerous machine learning reductions, importance weighting mechanisms, a variety of loss functions, and optimization techniques.
Notable features
VW encompasses support for:
- A broad spectrum of supervised and semi-supervised learning problems:
- Binary and multi-class classification
- Regression analysis
- Active learning applicable to both regression and classification tasks
- A diverse set of algorithms and model architectures:
- OLS regression
- Matrix factorization methods, including sparse matrix SVD
- Single-layer neural network models with adjustable hidden units
- Searn (Search and Learn) algorithm
- Latent Dirichlet Allocation (LDA)
- Stagewise polynomial approximations
- Top-K recommendation systems
- One-against-all (OAA) and cost-sensitive OAA strategies for multi-class classification
- Weighted all pairs approach
- Contextual bandit implementations featuring multiple exploration-exploitation strategies
- An array of loss functions:
- Squared error
- Quantile loss
- Hinge loss
- Logistic loss
- Poisson loss
- Optimization methods:
- Regularization techniques including (L1 norm, L2 norm, and elastic net regularization)
- Versatile input feature handling:
- Binary, numerical, and categorical data via feature hashing
- Effective management of sparse datasets and missing values
- Additional functionalities:
- Real-time feature interaction generation (quadratic, cubic)
- Dynamic creation of N-grams with optional skips tailored for textual data
- Automated test set holdout with early stopping across multiple iterations
- Support for bootstrapping procedures
- Customizable online progress tracking and model auditing tools
- Facilities for Hyperparameter optimization
Scalability
Vowpal Wabbit has showcased the capacity to handle datasets with up to one trillion features (1012) distributed over 1000 nodes within an hour. Its scalability is attributed to several core design elements:
- Out-of-core online learning that negates the necessity to load entire datasets into RAM
- Utilization of the hashing trick, which maps features to weight indices using 32-bit MurmurHash3 hashing
- A multi-threaded architecture that exploits multi-core processors by decoupling input parsing from the learning process
- An optimized implementation in compiled C++ for high efficiency