We introduce a linear programming-based approach for hyperparameter tuning of machine learning models. The approach finetunes continuous hyperparameters and model parameters through a linear program, enhancing model generalization in the vicinity of an initial model. The proposed method converts hyperparameter optimization into a bilevel program and identifies a descent direction to improve validation loss. The results demonstrate improvements in most cases across regression, machine learning, and deep learning tasks, with test performance enhancements ranging from 0.3% to 28.1%.