Minimalist Data Wrangling with Python

This is a draft of Minimalist Data Wrangling with Python by Marek Gagolewski. It’s distributed in the hope that it’ll be useful. Any bug/typos reports/fixes are appreciated. Although available online, this is a whole course, and should be read from the beginning to the end. In particular, refer to the Preface for general introductory remarks. Enjoy.

Copyright (C) 2022, Marek Gagolewski

This material is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0). This book is available both in PDF and HTML.

Start Here

• Preface
  • The Art of Data Wrangling
  • Aims and Scope
  • About the Author
  • Acknowledgements

Introducing Python

• 1. Getting Started with Python
  • 1.1. Installing Python
  • 1.2. Working with Jupyter Notebooks
  • 1.3. Initialising Each Session and Getting Example Data (!)
  • 1.4. Further Reading
  • 1.5. Exercises
• 2. Scalar Types and Control Structures in Python
  • 2.1. Scalar Types
  • 2.2. Calling Built-in Functions
  • 2.3. Controlling Program Flow
  • 2.4. Defining Own Functions
  • 2.5. Exercises
• 3. Sequential and Other Types in Python
  • 3.1. Sequential Types
  • 3.2. Working with Sequences
  • 3.3. Dictionaries (*)
  • 3.4. Iterable Types
  • 3.5. Object References and Copying (*)
  • 3.6. Further Reading
  • 3.7. Exercises

Unidimensional Data

• 4. Unidimensional Numeric Data and Their Distribution
  • 4.1. Creating Vectors in numpy
  • 4.2. Mathematical Notation
  • 4.3. Histograms
  • 4.4. Cumulative Counts and the Empirical Cumulative Distribution Function (*)
  • 4.5. Log-scale and Heavy-Tailed Distributions
  • 4.6. Exercises
• 5. Aggregating Numeric Data
  • 5.1. Measures of Location
  • 5.2. Measures of Dispersion
  • 5.3. Measures of Shape (*)
  • 5.4. Box (and Whisker) Plots
  • 5.5. Further Reading
  • 5.6. Exercises
• 6. Transforming and Filtering Numeric Data
  • 6.1. Vectorised Mathematical Functions
  • 6.2. Arithmetic Operators
  • 6.3. Vector Indexing
  • 6.4. Other Operations
  • 6.5. Exercises
• 7. Continuous Probability Distributions (*)
  • 7.1. Normal Distribution
  • 7.2. Assessing Goodness of Fit
  • 7.3. Other Noteworthy Distributions
  • 7.4. Generating Pseudorandom Numbers
  • 7.5. Exercises

Multidimensional Data

• 8. Multidimensional Numeric Data at a Glance
  • 8.1. Creating Matrices
  • 8.2. Reshaping Matrices
  • 8.3. Mathematical Notion
  • 8.4. Visualising Multidimensional Data
  • 8.5. Exercises
• 9. Transforming, Aggregating, and Filtering Multidimensional Data
  • 9.1. Vectorised Mathematical Functions
  • 9.2. Aggregation Functions
  • 9.3. Arithmetic, Logical, and Comparison Operations
  • 9.4. Other Row and Column Transforms (*)
  • 9.5. Matrix Indexing
  • 9.6. Exercises
• 10. 🚧 Exploring Relationships Between Variables
  • 10.1. Measuring Correlation
  • 10.2. Further Reading
  • 10.3. Exercises

Heterogeneous Data

• 11. Introducing Data Frames
  • 11.1. Creating Data Frames
  • 11.2. Aggregating
  • 11.3. Transforming
  • 11.4. Filtering Series Objects
  • 11.5. Filtering Data Frames
  • 11.6. Sorting
  • 11.7. Stacking and Unstacking
  • 11.8. Set-Theoretic Operations
  • 11.9. Joining (Merging)
  • 11.10. Further Reading
  • 11.11. Exercises
• 12. Handling Categorical Data
  • 12.1. Representing and Generating Categorical Data
  • 12.2. Frequency Distributions
  • 12.3. Visualising
  • 12.4. Aggregating
  • 12.5. Exercises
• 13. 🚧 Processing Data in Groups
  • 13.1. The groupby Methods
  • 13.2. Plotting Data in Groups
  • 13.3. Manual Splitting Into Subgroups (**)
  • 13.4. Exercises
• 14. Accessing Databases
  • 14.1. Example Database
  • 14.2. Exporting Data
  • 14.3. Exercises on SQL vs pandas
  • 14.4. Closing the Database Connection
  • 14.5. Common Data Serialisation Formats for the Web
  • 14.6. Working with Many Files
  • 14.7. Exercises

Text and Other Data

• 15. Working with Text Data
  • 15.1. String Operations in Base Python
  • 15.2. String Operations in pandas
  • 15.3. Working with String Lists
  • 15.4. String Operations in numpy (**)
  • 15.5. Formatted Outputs (*)
  • 15.6. Exercises
• 16. Regular Expressions (*)
  • 16.1. Regex Matching with re
  • 16.2. Regex Matching with pandas
  • 16.3. Matching Individual Characters
  • 16.4. Alternating and Grouping Subexpressions
  • 16.5. Quantifiers
  • 16.6. Capture Groups and References Thereto
  • 16.7. Anchoring
  • 16.8. Exercises
• 17. Outliers, Missing, Censored, and Incorrect Data
  • 17.1. Missing Data
  • 17.2. Incorrect Data
  • 17.3. Outliers
  • 17.4. Censored and Interval Data (*)
  • 17.5. Exercises
• 18. Time Series
  • 18.1. Temporal Ordering and Line Charts
  • 18.2. Working with Datetimes and Timedeltas
  • 18.3. Basic Operations
  • 18.4. Notes on Signal Processing (Audio, Image, Video) (**)
  • 18.5. Further Reading
  • 18.6. Exercises

Appendix

• Changelog
• Bibliography

Links

• This Book in PDF
• Report Bugs or Typos
• Datasets
• Author