Soft-SVM regression for binary classification and its extensions
OA Version
Citation
Abstract
My dissertation comprises three interconnected projects (Chapters 2 to 4), focused on developing a novel binary classification approach that blends elements of logistic regression and Support Vector Machines (SVMs), and extends this approach to other application tools such as a Bayesian and multi-class classifier. This innovative method is termed 'Soft-SVM' regression. Chapter 1 introduces the background information and the motivation for conducting research on this topic. It briefly summarizes the algorithms of logistic regression and SVM, as well as their corresponding strengths and limitations in performing binary classification. This discussion leads to our initial motivation for pursuing this research topic. Chapter 2 lays the groundwork by detailing the motivation and methodology behind Soft-SVM. Recognizing the similarities in the loss function shapes of logistic regression and SVM, we approach the problem through the lens of cumulant functions in the generalized linear model (GLM) framework. Soft-SVM is designed with shifted soft plus functions, allowing for a tailored balance between logistic regression and SVM characteristics. We employ Fisher scoring with active set method for constrained optimization to train parameters. The efficacy of Soft-SVM is demonstrated using simulated datasets and case studies, where its performance is benchmarked against traditional logistic regression and SVM. As we show, our proposed methodology offers better, more robust, and more computationally efficient performance. In Chapter 3, the narrative shifts to a Bayesian perspective on the same regression problem. Here, the regression coefficients and softness parameter are treated as random, while other aspects remain consistent with the developments in Chapter 2. We adopt the Riemannian manifold Hamiltonian Monte Carlo (RMHMC) method for parameter estimation, crafting a sampler for the posterior distribution of model parameters. Chapter 4 expands the scope of Soft-SVM by adapting it for multi-class classification scenarios. This adaptation opens up new potential for SVM in multi-class settings, representing a breakthrough that hasn't been previously studied.
Description
2024