CatBoost Predicts Fine-Tuning Time with R²=0.98 on H200 GPUs
⚡ Breaking News
Hugging Face
November 19, 20264 min read1

CatBoost Predicts Fine-Tuning Time with R²=0.98 on H200 GPUs

By — Senior AI Technical Editor

Back to News
❝

Dharma-AI's Hugging Face study of 272 production SFT runs on NVIDIA H200 GPUs (7,382 GPU-hours) shows traditional FLOPs estimation misses by 12.41 hours on average, while a CatBoost regression model predicts fine-tuning time with R²=0.98 and just 1.71 hours MAE before job launch.

Executive Overview

Dharma-AI has published a production-scale study on Hugging Face demonstrating that a CatBoost regression model can predict supervised fine-tuning (SFT) job duration on NVIDIA H200 GPUs with R² = 0.98 and a mean absolute error of just 1.71 hours—before the job is launched. The analysis covers 272 production runs totaling 7,382 GPU-hours, with job durations ranging from 1.6 to 172.4 hours (mean 23.43 hours, median 15.29 hours). By contrast, traditional FLOPs-based estimation at the best observed MFU of 30% yields R² = 0.31 and a mean error of 12.41 hours, making it unreliable for scheduling shared GPU clusters. The study is available immediately via the Hugging Face blog, with the full paper forthcoming on arXiv.

📊 Official Data & Technical Specifications Card

Technical AxisConfirmed Official Data
💰 Pricing & Usage CostThe study and research paper are completely free via the Hugging Face blog and arXiv; no subscription tiers or token pricing. Economic value is measured by saving wasted GPU-hours on shared H200 clusters.
🌐 Platforms & Immediate AvailabilityAvailable immediately on Hugging Face Blog (official link); full paper on arXiv soon. The technical stack relies on Hugging Face Transformers and Unsloth.
⚡ Performance & Speed BenchmarksCatBoost: R² = 0.98 ± 0.01, RMSE = 3.52 hours, MAE = 1.71 hours, median error 0.51 hours. XGBoost: R² = 0.97, MAE = 1.95 hours. Ridge: MAE = 7.23 hours. FLOPs (MFU 30%): R² = 0.31, MAE = 12.41 hours. Calculon: R² = −0.51, MAE = 20.36 hours.
🛡️ Security & Injection ResistanceThe report does not address security standards or prompt injection resistance; it focuses on resource scheduling and GPU utilization optimization in shared production environments.
🧠 Context WindowNot applicable; the study concerns training and fine-tuning time, not inference context window. Covered models reach up to 15 billion parameters.
🌍 Arabic Language & Regional SupportThe study is in English on Hugging Face; no official Arabic version. The methodology is reproducible on any Arabic production run logs provided similar data is available.

Deep-Dive Features & Architecture

The core innovation reframes fine-tuning time estimation from the theoretical FLOPs equation (6·P·D) into a supervised regression problem. A CatBoost model is trained on 24 features available before job launch, with no runtime logs required. These features include the number of evaluation samples (num_eval_data), number of checkpoints (num_saves), size of saved parameters (num_params_saved), tokens per GPU, and the FLOPs estimate itself at MFU = 0.45 as a primary input whose non-linear corrections the model learns. As a result, CatBoost captures non-linear interactions between periodic evaluation, checkpoint policy, per-GPU load, and tuning mode (LoRA vs. full fine-tuning)—signals a closed-form equation cannot see.

Feature importance analysis ranks num_eval_data first, followed by num_trainable_params, batch_eval_size_percent, num_all_params, and num_gpus×batch_train_percent, with num_params_saved close behind. SHAP values confirm that larger evaluation sets consistently push predicted time upward, while the effect of trainable parameter count is non-linear and only inflates for the largest models. In practice, errors across 272 out-of-sample predictions had a median of 0.51 hours, with 80% of predictions within 2.20 hours and 90% within 4.31 hours.

Benchmark & Competitive Performance

Under stratified five-fold cross-validation, CatBoost outperformed all alternatives: XGBoost achieved R² = 0.97 ± 0.01 and MAE = 1.95 ± 0.32 hours; Ridge reached R² = 0.73 and MAE = 7.23 hours; linear regression R² = 0.69 and MAE = 7.52 hours. Analytical models were far weaker: FLOPs estimation at 45% MFU scored R² = 0.18 and MAE = 13.01 hours, collapsing to R² = 0.01 and MAE = 14.62 hours at 60% MFU. The Calculon analytical model (Isaev et al., SC '23) estimated only 11% of the actual observed time. This sharp gap reflects that much of production job time lies outside the optimization loop itself: environment setup, data loading, and orchestration overhead.

Industry Impact & Enterprise Adoption

For enterprises running shared H200 clusters, the ability to predict fine-tuning duration with sub-2-hour accuracy before launch directly improves GPU scheduling, reduces idle time, and enables more reliable capacity planning. The model's reliance on pre-launch metadata—rather than runtime telemetry—makes it immediately deployable in multi-tenant environments where jobs are queued and resources are allocated in advance. Dharma-AI's release on Hugging Face, with the full paper forthcoming on arXiv, signals a shift toward data-driven scheduling for generative AI training workloads. The methodology is reproducible on any production logs with similar features, including Arabic-language operations, though no official Arabic version exists.

Conclusion

Dharma-AI's study provides a compelling, quantitative case for replacing FLOPs-based estimation with machine learning for fine-tuning time prediction. With R² = 0.98 and a 1.71-hour MAE on 272 real H200 runs, CatBoost offers a practical tool for GPU cluster management. While limited to SFT on H200 and not yet covering RLHF or other GPU generations, the approach is a significant step toward efficient, predictable AI training operations.

Media Source: Hugging Face | البيان الرسمي للشركة: المصدر الأصلي | Fact Verification & Analysis: AI Tools Oasis

Original Source:Hugging FaceThis news was formulated based on coverage from Hugging Face

Frequently Asked Questions

How accurate is CatBoost at predicting fine-tuning time compared to FLOPs estimation?

CatBoost achieves an R² of 0.98, a mean absolute error (MAE) of 1.71 hours, and a median error of 0.51 hours in five-fold cross-validation. In contrast, FLOPs estimation at 30% MFU yields only R²=0.31 and a 12.41-hour MAE across 272 real production runs.

How many runs and GPUs did the Dharma-AI study analyze?

The study analyzed 272 supervised fine-tuning (SFT) jobs on a production cluster of NVIDIA H200 GPUs, totaling 7,382 GPU-hours. Job durations ranged from 1.6 to 172.4 hours, with a mean of 23.43 hours and a median of 15.29 hours.

What is the best MFU value for FLOPs estimation and what error remains?

Scanning MFU values from 0.25 to 0.65, the best was 30%, yet the average error remained above 12 hours on jobs averaging 23 hours. This confirms that no single MFU value can correct FLOPs estimation.

What are the limitations of the announced fine-tuning time prediction model?

The model is limited to SFT jobs on H200 GPUs and does not cover RLHF, DPO, or RLVR. Its calibration is not expected to transfer to other GPU generations without retraining. Errors also widen above 60 hours due to sparse data (only 13 runs in that range).

Where can I follow the full research paper for the fine-tuning time prediction tool?

The full paper, titled 'Predicting End-to-End Fine-Tuning Time for Generative Language Models,' will be published on arXiv soon. You can follow Dharma-AI on Hugging Face via the official study link: https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time

AI Tools Oasis

AI Tools Oasis Team

Bringing you the latest news and analysis in the world of Artificial Intelligence with accuracy and credibility. Follow us for all updates.