
CatBoost Predicts Fine-Tuning Time with R²=0.98 on H200 GPUs
By Ali Al-Montaser — Senior AI Technical Editor
Dharma-AI's Hugging Face study of 272 production SFT runs on NVIDIA H200 GPUs (7,382 GPU-hours) shows traditional FLOPs estimation misses by 12.41 hours on average, while a CatBoost regression model predicts fine-tuning time with R²=0.98 and just 1.71 hours MAE before job launch.
Executive Overview
Dharma-AI has published a production-scale study on Hugging Face demonstrating that a CatBoost regression model can predict supervised fine-tuning (SFT) job duration on NVIDIA H200 GPUs with R² = 0.98 and a mean absolute error of just 1.71 hours—before the job is launched. The analysis covers 272 production runs totaling 7,382 GPU-hours, with job durations ranging from 1.6 to 172.4 hours (mean 23.43 hours, median 15.29 hours). By contrast, traditional FLOPs-based estimation at the best observed MFU of 30% yields R² = 0.31 and a mean error of 12.41 hours, making it unreliable for scheduling shared GPU clusters. The study is available immediately via the Hugging Face blog, with the full paper forthcoming on arXiv.
📊 Official Data & Technical Specifications Card
| Technical Axis | Confirmed Official Data |
|---|---|
| 💰 Pricing & Usage Cost | The study and research paper are completely free via the Hugging Face blog and arXiv; no subscription tiers or token pricing. Economic value is measured by saving wasted GPU-hours on shared H200 clusters. |
| 🌐 Platforms & Immediate Availability | Available immediately on Hugging Face Blog (official link); full paper on arXiv soon. The technical stack relies on Hugging Face Transformers and Unsloth. |
| ⚡ Performance & Speed Benchmarks | CatBoost: R² = 0.98 ± 0.01, RMSE = 3.52 hours, MAE = 1.71 hours, median error 0.51 hours. XGBoost: R² = 0.97, MAE = 1.95 hours. Ridge: MAE = 7.23 hours. FLOPs (MFU 30%): R² = 0.31, MAE = 12.41 hours. Calculon: R² = −0.51, MAE = 20.36 hours. |
| 🛡️ Security & Injection Resistance | The report does not address security standards or prompt injection resistance; it focuses on resource scheduling and GPU utilization optimization in shared production environments. |
| 🧠 Context Window | Not applicable; the study concerns training and fine-tuning time, not inference context window. Covered models reach up to 15 billion parameters. |
| 🌍 Arabic Language & Regional Support | The study is in English on Hugging Face; no official Arabic version. The methodology is reproducible on any Arabic production run logs provided similar data is available. |
Deep-Dive Features & Architecture
The core innovation reframes fine-tuning time estimation from the theoretical FLOPs equation (6·P·D) into a supervised regression problem. A CatBoost model is trained on 24 features available before job launch, with no runtime logs required. These features include the number of evaluation samples (num_eval_data), number of checkpoints (num_saves), size of saved parameters (num_params_saved), tokens per GPU, and the FLOPs estimate itself at MFU = 0.45 as a primary input whose non-linear corrections the model learns. As a result, CatBoost captures non-linear interactions between periodic evaluation, checkpoint policy, per-GPU load, and tuning mode (LoRA vs. full fine-tuning)—signals a closed-form equation cannot see.
Feature importance analysis ranks num_eval_data first, followed by num_trainable_params, batch_eval_size_percent, num_all_params, and num_gpus×batch_train_percent, with num_params_saved close behind. SHAP values confirm that larger evaluation sets consistently push predicted time upward, while the effect of trainable parameter count is non-linear and only inflates for the largest models. In practice, errors across 272 out-of-sample predictions had a median of 0.51 hours, with 80% of predictions within 2.20 hours and 90% within 4.31 hours.
Benchmark & Competitive Performance
Under stratified five-fold cross-validation, CatBoost outperformed all alternatives: XGBoost achieved R² = 0.97 ± 0.01 and MAE = 1.95 ± 0.32 hours; Ridge reached R² = 0.73 and MAE = 7.23 hours; linear regression R² = 0.69 and MAE = 7.52 hours. Analytical models were far weaker: FLOPs estimation at 45% MFU scored R² = 0.18 and MAE = 13.01 hours, collapsing to R² = 0.01 and MAE = 14.62 hours at 60% MFU. The Calculon analytical model (Isaev et al., SC '23) estimated only 11% of the actual observed time. This sharp gap reflects that much of production job time lies outside the optimization loop itself: environment setup, data loading, and orchestration overhead.
Industry Impact & Enterprise Adoption
For enterprises running shared H200 clusters, the ability to predict fine-tuning duration with sub-2-hour accuracy before launch directly improves GPU scheduling, reduces idle time, and enables more reliable capacity planning. The model's reliance on pre-launch metadata—rather than runtime telemetry—makes it immediately deployable in multi-tenant environments where jobs are queued and resources are allocated in advance. Dharma-AI's release on Hugging Face, with the full paper forthcoming on arXiv, signals a shift toward data-driven scheduling for generative AI training workloads. The methodology is reproducible on any production logs with similar features, including Arabic-language operations, though no official Arabic version exists.
Conclusion
Dharma-AI's study provides a compelling, quantitative case for replacing FLOPs-based estimation with machine learning for fine-tuning time prediction. With R² = 0.98 and a 1.71-hour MAE on 272 real H200 runs, CatBoost offers a practical tool for GPU cluster management. While limited to SFT on H200 and not yet covering RLHF or other GPU generations, the approach is a significant step toward efficient, predictable AI training operations.
Media Source: Hugging Face | البيان الرسمي للشركة: المصدر الأصلي | Fact Verification & Analysis: AI Tools Oasis
Frequently Asked Questions
CatBoost achieves an R² of 0.98, a mean absolute error (MAE) of 1.71 hours, and a median error of 0.51 hours in five-fold cross-validation. In contrast, FLOPs estimation at 30% MFU yields only R²=0.31 and a 12.41-hour MAE across 272 real production runs.
The study analyzed 272 supervised fine-tuning (SFT) jobs on a production cluster of NVIDIA H200 GPUs, totaling 7,382 GPU-hours. Job durations ranged from 1.6 to 172.4 hours, with a mean of 23.43 hours and a median of 15.29 hours.
Scanning MFU values from 0.25 to 0.65, the best was 30%, yet the average error remained above 12 hours on jobs averaging 23 hours. This confirms that no single MFU value can correct FLOPs estimation.
The model is limited to SFT jobs on H200 GPUs and does not cover RLHF, DPO, or RLVR. Its calibration is not expected to transfer to other GPU generations without retraining. Errors also widen above 60 hours due to sparse data (only 13 runs in that range).
The full paper, titled 'Predicting End-to-End Fine-Tuning Time for Generative Language Models,' will be published on arXiv soon. You can follow Dharma-AI on Hugging Face via the official study link: https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time

AI Tools Oasis Team
Bringing you the latest news and analysis in the world of Artificial Intelligence with accuracy and credibility. Follow us for all updates.
