How do you assess the data quality using python?

 

There is an amazing open-source python library ‘ydata_quality . It assesses Data Quality throughout the multiple stages of a data pipeline development.

Once you have a dataset available, running DataQuality(df=my_df).evaluate() provides a comprehensive overview of the details and intricacies of the data, through the perspective of the multiple modules available in the package.

The library focuses on the following individual modules as well.

1.     Bias and Fairness

Checks the bias and fairness in the dataset.

Ø  Bias: A systematic, non-neglectable treatment which is differentiated towards a specific sub-group of individuals

Ø  Fairness: The absence of differentiated treatment (assistive or punitive) based on sensitive attributes. Fairness can also be thought of as the absence of unjustified basis for differentiated treatment.

2.     Data Expectations

To define an expectation about data is to develop a unit test that asserts a certain property about the data and provides an actionable output in any deviation.

3.     Data Relations

Checks the data correlation and causality between variables in the dataset. This library implements the checks by leveraging techniques such as correlation methods, full order partial correlations (i.e. controlling the effect of all covariates) and with Variance Inflation Factor, which is used to assess multicollinearity of numerical covariates.

4.     Drift Analysis

Data drift is a broad term used for differences in the data observed by a model during training and prediction time. The library uses different methods like Maximum Mean Discrepancy, Kolmogorov-Smirnov test, Chi-Squared test and Bonferroni correction depends on the type of data.

5.     Duplicates

Functionality to detect duplicate values.

6.     Labelling: Categoricals and Numericals

Detects the quality issues such as imbalanced class, missing labels, outlier detection, etc in the dataset based on the type of labels.

7.     Missings

Detects null/missing information in your dataset

8.     Erroneous Data

Functionality to detect the erroneous data values in the data frame.


Source:https://github.com/ydataai/ydata-quality

Comments

Post a Comment

Popular posts from this blog

How do you build a data warehouse in SQL server?

Building a data warehouse in SQL server is a complex process that requires a deep understanding of SQL Server and data warehousing best practices. It involves several steps, including planning, designing, and implementing the database schema, as well as loading and transforming the data. Here are the high-level steps you can follow to build a data warehouse in SQL Server: ·          Identify the business requirements : Before you start building a data warehouse, you need to identify the business requirements, such as what data is needed, who will use the data, and how the data will be analysed. ·         Plan the data warehouse schema: Once you have identified the business requirements, you need to plan the data warehouse schema. This involves designing the tables, relationships, and indexes that will be used to store the data. ·         Create the database: After plann...
Read more

When to choose data warehouse over database?

  The decision between a database and a data warehouse will depend on your specific needs and use case. Deciding between a database and a data warehouse largely depends on the specific needs of your organization and the type of data you're working with. Some of the factors for consideration are: Ø  Data Volume : When you have large volume of data (e.g., terabytes or more), then better to go for a data warehouse as data warehouses are optimized for handling large amounts of data. Databases are designed to handle smaller amounts of data. Ø  Data Complexity : If you are working with complex data that requires advanced querying and analysis, then, a data warehouse will be a better choice. Data warehouses can handle complex queries and data modelling. Databases may not be equipped to handle the same level of complexity. Ø  Data Integration : If you need to integrate data from multiple sources, a data warehouse may be the better choice. Data warehouses are designed to hand...
Read more