AccScience Publishing / IJOCTA / Online First / DOI: 10.36922/IJOCTA026200080
Submit to IJOCTA
Cite this article
9
Download
100
Views
Related Info Links
Google Scholar
More by Authors Links
Adriana Florescu
Journal Browser
Volume | Year
Issue
Search
Forthcoming Issue
Current Issue
View All
News and Announcements
View All
RESEARCH ARTICLE

Resampling strategies for machine learning-based effort overrun risk detection: A controlled factorial study with cost-sensitive evaluation

Andreea-Elena Catana1 ,  Adriana Florescu1*
Show Less
1 Department of Engineering and Industrial Management, Faculty of Technological Engineering and Industrial Management, Transilvania University of Brasov, Brasov, Romania
IJOCTA, 026200080 https://doi.org/10.36922/IJOCTA026200080
Received: 11 May 2026 | Revised: 2 June 2026 | Accepted: 9 June 2026 | Published online: 9 July 2026
© 2026 by the Author(s). This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution -Noncommercial 4.0 International License (CC-by the license) ( https://creativecommons.org/licenses/by-nc/4.0/ )
Download PDF
XML
Cite
Abstract

The Synthetic Minority Oversampling Technique (SMOTE) has become the default preprocessing step for handling class imbalance in software effort-risk prediction, yet its effectiveness in this domain has not been rigorously tested. This study addresses that gap through a controlled factorial experiment on the publicly available Software Development Effort Dataset Annotated with Expert Estimates dataset, comprising 4,329 software issues from Apache projects drawn from an initial repository of 23,186 records. Six resampling strategies are compared across four classifier families under both standard and cost-sensitive evaluation metrics that weight missed high-risk issues more heavily than false alarms. A secondary contribution is the analysis of how resampling interacts with the extreme class imbalance characteristic of real-world effort data (2.2% minority rate), a regime substantially more severe than those examined in prior investigations. Results are interpreted through SHAP-based feature attribution to determine whether oversampling alters which features the models rely on. The findings reveal two overarching results: near-perfect performance under a full feature set is largely attributable to target leakage rather than a genuine predictive signal, and, under deployment, valid early-warning features degrade cost-sensitive performance relative to no resampling when SMOTE is used. Cost-sensitive weighting emerges as the more reliable alternative, preserving both performance and feature attribution structure. These findings challenge the uncritical adoption of SMOTE in software analytics and carry direct implications for the design of reproducible, interpretable risk-detection pipelines.

Keywords
Software effort estimation
Machine learning
Software analytics
Risk detection
Class imbalance
Funding
This study received no external funding. The publication fee for the article was borne by Transilvania University of Brașov.
Conflict of interest
The authors declare no conflict of interest.
References
  1. Tran TN, Tran HT, Nguyen QN. Leveraging AI for Enhanced Software Effort Estimation: A Comprehensive Study and Framework Proposal. In: Proceedings of the 2023 International Conference on Cognitive Computing and Complex Data (ICCD), Huaian, China, 2023:284–289. https://doi.org/10.1109/ICCD59681.2023.10420603
  2. Alhamed M, Storer T. Evaluation of Context- Aware Language Models and Experts for Effort Estimation of Software Maintenance Issues. In: Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), Limassol, Cyprus, 2022:129–138. https://doi.org/10.1109/ICSME55016.2022.00020
  3. Kotsiantis S, Kanellopoulos D, Pintelas P. Handling imbalanced datasets: A review. GESTS Int Trans Comput Sci Eng. 2006;30:25–36. April 6, 2026. https://www.researchgate.net/ publication/228084509_Handling_imbalanced_datasets_A_review
  4. He H, Garcia EA. Learning from Imbalanced Data. IEEE Trans Knowl Data Eng. 2009;21(9):1263– 1284. https://doi.org/10.1109/TKDE.2008.239
  5. Fernández A, García S, Galar M, Prati RC, Krawczyk B, Herrera F. Learning from Imbalanced Data Sets. Cham: Springer; 2018:19-46. https://doi.org/10.1007/978-3-319-98074-4
  6. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. J Artif Intell Res. 2002;16:321–357. https://doi.org/10.1613/jair.953
  7. Song Q, Guo Y, Shepperd M. A Comprehensive Investigation of the Role of Imbalanced Learning for Software Defect Prediction. IEEE Tran Softw Eng. 2019;45(12):1253–1269. https://doi.org/10.1109/TSE.2018.2836442
  8. Jude A, Uddin J. Explainable Software Defects Classification Using SMOTE and Machine Learning. Ann Emerg Technol Comput. 2024;8(1):36–49. https://doi.org/10.33166/AETiC.2024.01.004
  9. Ali SS, Ren J, Zhang K, Wu J, Liu C. Heterogeneous Ensemble Model to Optimize Software Effort Estimation Accuracy. IEEE Access. 2023;11:27759–27792. https://doi.org/10.1109/ACCESS.2023.3256533
  10. ForouzeshNejad AA, Arabikhan F, Aheleroff S. Optimizing Project Time and Cost Prediction Using a Hybrid XGBoost and Simulated Annealing Algorithm. Machines. 2024;12(12):867. https://doi.org/10.3390/machines12120867
  11. Ferhati K, Burlea-Schiopoiu A, Nascu AG. A Text-Based Project Risk Classification System Using Multi-Model AI: Comparing SVM, Logistic Regression, Random Forests, Naive Bayes, and XGBoost. Systems. 2025;13(12):1078. https://doi.org/10.3390/systems13121078
  12. Fernández A, García S, Herrera F, Chawla NV. SMOTE for learning from imbalanced data: progress and challenges, marking the 15-year anniversary. J Artif Intell Res. 2018;61:863–905. https://doi.org/10.1613/jair.1.11192
  13. Aflaha R, Herteno R, Faisal MR, Abadi F, Saputro S. Effect of SMOTE Variants on Software Defect Prediction Classification Based on Boosting Algorithm. J Ilm Tek Elektro Komput Inform. 2024;10(2):201–216. https://doi.org/10.26555/jiteki.v10i2.28521
  14. Imani M, Beikmohammadi A, Arabnia HR. Comprehensive Analysis of Random Forest and XGBoost Performance with SMOTE, ADASYN, and GNUS Under Varying Imbalance Levels. Technologies. 2025;13(3):88. https://doi.org/10.3390/technologies13030088
  15. Alhamed M, Storer T. JOSSE: A Software Development Effort Dataset Annotated with Expert Estimates, Version 1. 2025. https://doi.org/10.5281/zenodo.7022735
  16. Mende T, Koschke R. Revisiting the evaluation of defect prediction models. In: Promise’09: Proceedings of the 5th International Conference on Predictor Models in Software Engineering, New York, NY, USA:ACM; 2009; 7, 1–10. Accessed April 21, 2026. https://promisedata. org/pdf/2009/06_Mende.pdf
  17. Hall T, Beecham S, Bowes D, Gray D, Counsell S. A Systematic Literature Review on Fault Prediction Performance in Software Engineering. IEEE Trans Softw Eng. 2012;38(6):1276–1304. https://doi.org/10.1109/TSE.2011.103
  18. Blagus R, Lusa L. SMOTE for high-dimensional class-imbalanced data. BMC Bioinform. 2013;14(1):106. https://doi.org/10.1186/1471-2105-14-64
  19. Kovács, G. SMOTE-variants: A Python implementation of 85 minority oversampling techniques. Neurocomputing. 2019;366:352–360. https://doi.org/10.1016/j.neucom.2019.06.100
  20. Han H, Wang WY, Mao BH. Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning. In Advances in Intelligent Computing (ICIC), Lecture Notes in Computer Science, Berlin: Springer; 2005;3644:878–887. https://doi.org/10.1007/11538059_91
  21. He H, Bai Y, Garcia EA, Li S. ADASYN: Adaptive Synthetic Sampling Approach for Imbalanced Learning. In: Proceedings of the IEEE International Joint Conference on Neural Networks, Hong Kong: IEEE; 2008:1322–1328. https://doi.org/10.1109/IJCNN.2008.4633969
  22. Nguyen HM, Cooper EW, Kamei K. Borderline over-sampling for imbalanced data classification. Int J Knowl Eng Soft Data Paradig. 2011;3(1):4– 21. https://doi.org/10.1504/IJKESDP.2011.039875
  23. Ghinaya H, Hereno R, Faisal MR, Farmadi A, Indriani F. Analysis of Important Features in Software Defect Prediction Using SMOTE, RFE and Random Forest. J Electron Electromed Eng Med Inform. 2024;6(3):276-288. https://doi.org/10.35882/jeeemi.v6i3.453
  24. Jørgensen M. A review of studies on expert estimation of software development effort. J Syst Softw. 2014;70(1–2):37–60. https://doi.org/10.1016/S0164-1212(02)00156-5
  25. Idri A, Hosni M, Abran A. Systematic mapping study of ensemble effort estimation. J Inf Softw Technol.2016;118:151-175. https://doi.org/10.1016/j.jss.2016.05.016
  26. Herbold S. On the costs and profit of software defect prediction. IEEE Trans Softw Eng. 2021;47(11):2617–2631. https://doi.org/10.1109/TSE.2019.2957794
  27. Wen J, Li S, Lin Z, Hu Y, Huang C. Systematic literature review of machine learning based software development effort estimation models. Inf Softw Technol. 2012;54(1):41–59. https://doi.org/10.1016/j.infsof.2011.09.002
  28. Jadhav A, Shandilya K. Reliable machine learning models for estimating effective software development efforts: A comparative analysis. J Eng Res. 2023;11(4):362–376. https://doi.org/10.1016/j.jer.2023.100150
  29. Malhotra R, Jain A. Software effort prediction using statistical and machine learning methods. Int J Adv Comput Sci Appl. 2011;2(1). https://doi.org/10.14569/IJACSA.2011.020122
  30. Boehm BW, Abts C, Brown AW, et al. Software Cost Estimation with COCOMO II; Upper Saddle River, NJ, USA: Prentice Hall; 2009. https:// dl.acm.org/doi/10.5555/1795822
  31. Kumar PS, Behera HS. Estimating software effort using neural network: An experimental investigation. In Computational Intelligence in Pattern Recognition; Singapore: Springer; 2020:1120:165–180. https://doi.org/10.1007/978-981-15-2449-3_14
  32. Weinan T. Application of support vector machine system introducing multiple submodels in data mining. Syst Soft Comput. 2024;6:200096. https://doi.org/10.1016/j.sasc.2024.200096
  33. Matloob F, Ghazal T, Taleb N. Software Defect Prediction using Ensemble Learning: A Systematic Literature Review. IEEE Access. 2021;9:98754– 98771. https://doi.org/10.1109/ACCESS.2021.3095559
  34. Kumar H, Saxena V. Software Defect Prediction Using Hybrid Machine Learning Techniques: A Comparative Study. J Softw Eng Appl. 2024;17(4):155–171. https://doi.org/10.4236/jsea.2024.174009
  35. Ke G, Meng Q, Finley T, et al. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Red Hook, NY, USA: Curran Associates Inc.; 2017:3149–3157. https://doi.org/10.52202/068431-1367
  36. Prokhorenkova L, Gusev G, Vorobev A, Dorogush AV, Gulin A. CatBoost: unbiased boosting with categorical features. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS’18). Red Hook, NY, USA: Curran Associates Inc.; 2018:6639–6649. https://dl.acm.org/doi/ abs/10.5555/3327757.3327770
  37. Gao K, Khoshgoftaar TM, Wang H, Seliya N. Choosing software metrics for defect prediction: an investigation on feature selection techniques. Softw PractExp. 2011;41(5):579–606. https://doi.org/10.1002/spe.1043
  38. Drummond C, Holte RC. Cost Curves: An Improved Method for Visualizing Classifier Performance. Mach Learn. 2006;65(1):95–130. https://doi.org/10.1007/s10994-006-8199-5
  39. Lundberg SM, Lee SI. A Unified Approach to Interpreting Model Predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Red Hook, NY, USA: Curran Associates Inc.; 2017:4768–4777. Accessed April 18, 2026. https:// dl.acm.org/doi/10.5555/3295222.3295230
  40. Demšar J. Statistical Comparisons of Classifiers over Multiple Data Sets. J Mach Learn Res. 2006;7:1–30. Accessed April 18, 2026. https:// jmlr.org/papers/volume7/demsar06a/demsar06a. pdf
  41. Romano J, Kromrey JD, Coraggio J, Skowronek J. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’s d for evaluating group differences on the NSSE and similar surveys? In: Proceedings of the Annual Meeting of the Florida Association of Institutional Research, Cocoa Beach, FL, USA. 2006:1-3. Accessed April 21, 2026. https://www.researchgate.net/publication/237544991_Appropriate_Statistics_for_Ordinal_Level_Data_Should_We_Really_Be_Using_t-test_and_Cohen’s_d_for_Evaluating_Group_Differences_on_the_NSSE_and_other_Surveys
  42. Hosseini S, Turhan B, Gunarathna D. A Systematic Literature Review and Meta-Analysis on Cross Project Defect Prediction. IEEE Trans Softw Eng. 2019;45(2):111–147. https://doi.org/10.1109/TSE.2017.2770124
  43. Shepperd M, Bowes D, Hall T. Researcher Bias: The Use of Machine Learning in Software Defect Prediction. IEEE Trans Softw Eng. 2014;40(6):603–616. https://doi.org/10.1109/TSE.2014.2322358
Share
Back to top
An International Journal of Optimization and Control: Theories & Applications, Electronic ISSN: 2146-5703 Print ISSN: 2146-0957, Published by AccScience Publishing
We use cookies on our website to ensure you get the best experience.
Read more about our cookies here
Accept