Minimalist Data Wrangling with Python

Minimalist Data Wrangling with Python is envisaged as a student’s first introduction to data science, providing a high-level overview as well as discussing key concepts in detail. We explore methods for cleaning data gathered from different sources, transforming, selecting, and extracting features, performing exploratory data analysis and dimensionality reduction, identifying naturally occurring data clusters, modelling patterns in data, comparing data between groups, and reporting the results.

For many students around the world, educational resources are hardly affordable. Therefore, I have decided that this book should remain an independent, non-profit, open-access project (available both in PDF and HTML forms). Whilst, for some people, the presence of a “designer tag” from a major publisher might still be a proxy for quality, it is my hope that this publication will prove useful to those who seek knowledge for knowledge’s sake.

Please spread the news about it by sharing the above URLs with your mates, peers, or students. Thank you.

Also, consider citing this book as: Gagolewski M. (2022), Minimalist Data Wrangling with Python, Zenodo, Melbourne, DOI: 10.5281/zenodo.6451068, ISBN: 978-0-6455719-1-2, URL: https://datawranglingpy.gagolewski.com/.

A printed version (the same as the aforementioned PDF one) can be ordered from Amazon: AU CA DE ES FR IT JP NL PL SE UK US. Note that I get 0% revenue from sales (price = cost of printing + Amazon fee). If you know a vendor who can deliver the book to some geographic regions more cheaply, let me know.

Any bug/typo reports/fixes are appreciated. Although available online, this is a whole course, and should be read from the beginning to the end. In particular, refer to the Preface for general introductory remarks.

Copyright (C) 2022 by Marek Gagolewski. Some rights reserved.

This material is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

Start Here

• Preface
  • The Art of Data Wrangling
  • Aims, Scope, and Design Philosophy
  • Structure
  • The Rules
  • About the Author
  • Acknowledgements

Introducing Python

• 1. Getting Started with Python
  • 1.1. Installing Python
  • 1.2. Working with Jupyter Notebooks
  • 1.3. The Best Note-Taking App
  • 1.4. Initialising Each Session and Getting Example Data (!)
  • 1.5. Exercises
• 2. Scalar Types and Control Structures in Python
  • 2.1. Scalar Types
  • 2.2. Calling Built-in Functions
  • 2.3. Controlling Program Flow
  • 2.4. Defining Own Functions
  • 2.5. Exercises
• 3. Sequential and Other Types in Python
  • 3.1. Sequential Types
  • 3.2. Working with Sequences
  • 3.3. Dictionaries
  • 3.4. Iterable Types
  • 3.5. Object References and Copying (*)
  • 3.6. Further Reading
  • 3.7. Exercises

Unidimensional Data

• 4. Unidimensional Numeric Data and Their Empirical Distribution
  • 4.1. Creating Vectors in numpy
  • 4.2. Mathematical Notation
  • 4.3. Inspecting the Data Distribution with Histograms
  • 4.4. Exercises
• 5. Processing Unidimensional Data
  • 5.1. Aggregating Numeric Data
  • 5.2. Vectorised Mathematical Functions
  • 5.3. Arithmetic Operators
  • 5.4. Indexing Vectors
  • 5.5. Other Operations
  • 5.6. Exercises
• 6. Continuous Probability Distributions
  • 6.1. Normal Distribution
  • 6.2. Assessing Goodness-of-Fit
  • 6.3. Other Noteworthy Distributions
  • 6.4. Generating Pseudorandom Numbers
  • 6.5. Further Reading
  • 6.6. Exercises

Multidimensional Data

• 7. Multidimensional Numeric Data at a Glance
  • 7.1. Creating Matrices
  • 7.2. Reshaping Matrices
  • 7.3. Mathematical Notation
  • 7.4. Visualising Multidimensional Data
  • 7.5. Exercises
• 8. Processing Multidimensional Data
  • 8.1. From Vectors to Matrices
  • 8.2. Indexing Matrices
  • 8.3. Matrix Multiplication, Dot Products, and the Euclidean Norm
  • 8.4. Pairwise Distances and Related Methods
  • 8.5. Exercises
• 9. Exploring Relationships Between Variables
  • 9.1. Measuring Correlation
  • 9.2. Regression Tasks
  • 9.3. Finding Interesting Combinations of Variables (*)
  • 9.4. Further Reading
  • 9.5. Exercises

Heterogeneous Data

• 10. Introducing Data Frames
  • 10.1. Creating Data Frames
  • 10.2. Aggregating Data Frames
  • 10.3. Transforming Data Frames
  • 10.4. Indexing Series Objects
  • 10.5. Indexing Data Frames
  • 10.6. Further Operations on Data Frames
  • 10.7. Exercises
• 11. Handling Categorical Data
  • 11.1. Representing and Generating Categorical Data
  • 11.2. Frequency Distributions
  • 11.3. Visualising Factors
  • 11.4. Aggregating and Comparing Factors
  • 11.5. Exercises
• 12. Processing Data in Groups
  • 12.1. Basic Methods
  • 12.2. Plotting Data in Groups
  • 12.3. Classification Tasks
  • 12.4. Clustering Tasks
  • 12.5. Further Reading
  • 12.6. Exercises
• 13. Accessing Databases
  • 13.1. Example Database
  • 13.2. Exporting Data to a Database
  • 13.3. Exercises on SQL vs pandas
  • 13.4. Closing the Database Connection
  • 13.5. Common Data Serialisation Formats for the Web
  • 13.6. Working with Many Files
  • 13.7. Exercises

Other Data Types

• 14. Text Data
  • 14.1. Basic String Operations
  • 14.2. Working with String Lists
  • 14.3. Formatted Outputs for Reproducible Report Generation
  • 14.4. Regular Expressions (*)
  • 14.5. Exercises
• 15. Missing, Censored, and Questionable Data
  • 15.1. Missing Data
  • 15.2. Censored and Interval Data (*)
  • 15.3. Incorrect Data
  • 15.4. Outliers
  • 15.5. Exercises
• 16. Time Series
  • 16.1. Temporal Ordering and Line Charts
  • 16.2. Working with Datetimes and Timedeltas
  • 16.3. Basic Operations
  • 16.4. Further Reading
  • 16.5. Exercises

Appendix

• Changelog
• References