Aggregations such as sum and mean are frequently used in data science and analysis. There are cases where you might want to aggregate values over columns instead of rows. Meaning that if you have two columns A and B, you want to aggregate values on horizontally (columns), not vertically (rows). Polars allows this type of…
Month: June 2023
DuckDB with Polars, Pandas, and Arrow
One of the features in DuckDB is its integration with other data libraries such as pandas. DuckDB makes it seamless when we convert to and from other dataframes and table formats. This flexibility gives the users the ability to implement DuckDB in their data pipelines with ease. In this post, I’ll walk you through how…
Read from and Write to Amazon S3 in Polars
How do you work with Amazon S3 in Polars? Amazon S3 bucket is one of the most common object stores for data projects. Polars being a fairly new technology, there is not a ton of resources that explain how to work with S3. In this post, I’ll walk you through reading from and writing to…
Handling Missing Values in Polars
Checking and filling missing values is an important piece in data science and analytics projects. A popular dataframe library, pandas, provides a method like fillna(), for example. Polars has built-in methods and expressions to work with missing values as well. This post covers ways to check missing values as well as ways how you can…
LazyFrame vs DataFrame in Polars – Performance Comparison
One of the features in Polars is LazyFrame. Polars is fast as is, and LazyFrame gives you even more optimizations. But you may wonder, “How is it different from the typical DataFrame or EagerFrame?” or “What is LazyFrame in the first place”? What is LazyFrame in Polars In order to understand LazyFrame, it’s good to…
Group Rows into List in Polars
I recently encountered a situation where I wanted to consolidate or group rows per group value into a Python list. There seems to be various solutions in pandas (a few resources at the bottom), but how can you do this in Polars? There are probably multiple ways you can do it in Polars as well….