Improving Dataset Integrity Through Automated Data Cleaning Techniques

Year : 2026 | Volume : 04 | Issue : 01 | Page : 40 45
    By

    Rajani Kanta Sahu,

  • Achinta Kumar Palit,

  • Ushashree Nayak,

  1. Assistant Professor, Department of Computer science and Engineering, Gandhi Institute of Excellent Technocrats, Bhubaneswar, Odissa, India
  2. Assistant Professor, Department of Computer science and Engineering, Gandhi Institute of Excellent Technocrats, Bhubaneswar, Odissa, India
  3. Student, Department of Computer science and Engineering, Gandhi Institute of Excellent Technocrats, Bhubaneswar, Odissa, India

Abstract

High-quality data is a fundamental requirement in data science for producing trustworthy analytical insights and effective machine learning models. Problems, including incomplete records, inconsistent entries, duplicate observations, and anomalous values, can severely reduce the accuracy and robustness of predictive systems. As modern datasets continue to expand in both volume and structural complexity, relying on manual data cleaning methods become time-consuming and error-prone, highlighting the growing importance of automated data preprocessing solutions. This paper explores the automation of data cleaning within machine learning workflows, reviewing tools such as Open Refine, Trifacta, Data Cleaner, and Python libraries like Pandas and Dedupe. It examines key algorithmic approaches, including imputation techniques, to find unusual or abnormal data points (e.g., Z-score, interquartile range (IQR), Isolation Forest) and duplicate detection via fuzzy matching. The role of data profiling and validation frameworks in early-stage diagnosis is also discussed. Furthermore, the paper investigates how machine learning can be applied to data cleaning itself, enabling models to learn patterns for anomaly detection and correction. Challenges such as scalability, interpretability, and domain-specific customization are addressed, alongside future directions for autonomous data quality management. Interactive platforms like Data Lens, which combine profiling, error detection, and human-in-the-loop repair mechanisms, are highlighted for their role in building reproducible, ML-aligned workflows. The findings emphasize the transformative potential of AI-powered data cleaning in improving data integrity while reducing manual effort and operational cost.

Keywords: Data quality, data cleaning automation, machine learning workflows, outlier detection, imputation techniques, duplicate detection, data profiling

[This article belongs to International Journal of Data Structure Studies ]

How to cite this article:
Rajani Kanta Sahu, Achinta Kumar Palit, Ushashree Nayak. Improving Dataset Integrity Through Automated Data Cleaning Techniques. International Journal of Data Structure Studies. 2026; 04(01):40-45.
How to cite this URL:
Rajani Kanta Sahu, Achinta Kumar Palit, Ushashree Nayak. Improving Dataset Integrity Through Automated Data Cleaning Techniques. International Journal of Data Structure Studies. 2026; 04(01):40-45. Available from: https://journals.stmjournals.com/ijdss/article=2026/view=246381


References

  1. Dasu T, Johnson T. Exploratory Data Mining and Data Cleaning. Hoboken (NJ, US): Wiley Interscience; 2003. doi:10.1002/0471448354.
  2. Aggarwal CC. Outlier Analysis. 2nd ed. Cham: Springer; 2017. doi:10.1007/978-3-319-47578-3.
  3. Rahm E, Do HH. Data cleaning: problems and current approaches. IEEE Data Eng Bull. 2000;23(4):3–13.
  4. Redman TC. The impact of poor data quality on the typical enterprise. Commun ACM. 1998;41(2):79–82. doi:10.1145/269012.269025.
  5. io. (2023). Dedupe.io – De-duplicate and find matches in your Excel spreadsheet or database. [online] Dedupe.io. Available at: https://dedupe.io/
  6. Pipino LL, Lee YW, Wang RY. Data quality assessment. Commun ACM. 2002;45:211–218. doi:10.1145/505248.506010.
  7. (2026). OpenRefine: a powerful free, open source tool. [online] OpenRefine. Available from: https://openrefine.org/
  8. PyData Sphinx Theme. (2026). Pandas Documentation — Pandas 3.0.2 documentation. Version: 3.0.2. [online] Pandas: Open Source, BSD-Licensed Library. Available from: https://pandas.pydata.org/docs/
  9. Scikit-learn. (2025). Scikit-Learn: machine learning in Python — scikit-learn 1.8.0 documentation. [online] Available from: https://scikit-learn.org/stable/
  10. (2023). Trifacta is Now Alteryx Designer Cloud. [online] Alteryx. Available from: https://www.alteryx.com/about-us/trifacta-is-now-alteryx-designer-cloud
  11. Van der Walt S, Colbert SC, Varoquaux G. The NumPy array: a structure for efficient numerical computation. Comput Sci Eng. 2011;13(2):22–30. doi:10.1109/MCSE.2011.37.
  12. McKinney W. Data structures for statistical computing in Python. Python in Science Conference, Austin, Texas, USA. 2010. p. 56–61. doi:10.25080/Majora-92bf1922-00a.
  13. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–2830.
  14. Wickham H. Tidy data. J Stat Softw. 2014;59(10):1–23. doi:10.18637/jss.v059.i10.
  15. Chu X, Ilyas IF, Krishnan S, Wang J. Data cleaning: overview and emerging challenges. International Conference on Management of Data (SIGMOD/PODS’16), San Francisco, California, USA. 2016 Jun 26–Jul 1. p. 2201–2206. doi:10.1145/2882903.2912574.
  16. Bertrand Cariou (2020). New Dataprep AI features for data wrangling. [online] Google Cloud Blog. Available from: https://cloud.google.com/blog/products/data-analytics/new-dataprep-ai-features-for-data-wrangling
  17. Kandel S, Paepcke A, Hellerstein J, Heer J. Wrangler: interactive visual specification of data transformation scripts. CHI Conference on Human Factors in Computing Systems (CHI ‘11), Vancouver, BC, Canada. 2011 May 7–12. p. 3363–3372. doi:10.1145/1978942.1979444.
  18. Bantilan N. Pandera: statistical data validation of pandas dataframes. Proc Python Sci Conf. 2020:116–124. doi:10.25080/Majora-342d178e-010.
  19. Python Cerberus. (2023). Validation Schemas. [online] Cerberus — Data validation for Python. Available from: https://docs.python-cerberus.org/schemas.html
  20. Hershberg T, Burstein A, Dockhorn R. Record linkage. Hist Methods Newsl. 1976;9(2-3):137–163. doi:10.1080/00182494.1976.10112639
  21. (2020). FuzzyWuzzy: fuzzy string matching in Python [Online]. GitHub. Available from: https://github.com/seatgeek/fuzzywuzzy
  22. Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York (NY, US): John Wiley & Sons; 1987. doi:10.1002/9780470316696.
  23. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res. 2002;16:321–357. doi:10.1613/jair.953.
Regular Issue Subscription Review Article
Volume 04
Issue 01
Received 27/02/2026
Accepted 06/04/2026
Published 07/05/2026
Publication Time 69 Days
3

Login


My IP

PlumX Metrics