So when you create a brand new aws glue job, I don’t know about you but it seems pretty intimidating that there are 6 python import statements that are generated automatically. …

There are a lot more people that know SQL than know how to program in python and are proficient in Spark to perform big data analytics on their data. With AWS Glue Studio, it’s possible to build data pipelines for big data analytics on a distributed cluster without knowing to…

How to create, insert, and query records in your governed table using the AWS Data Wrangler library in Python — Governed Tables, is a new type of table on Amazon S3 that supports ACID transactions which can help you build more resilient data pipelines for your data lake on AWS. Governed tables can be enabled on top of your data in AWS S3 in the following formats: Avro, CSV, JSON…

When it comes to data analytics and building data pipelines in python, you have probably heard of pandas which is versatile for data analytics on a single machine and Pyspark for data analytics in a distributed environment when you are working with too much data for one machine to handle…

Pandas is an extremely popular and essential python package for data science as it’s powerful, flexible and easy to use open-source data analysis and data manipulation. …

Parquet is an open-sourced columnar storage format created by the Apache software foundation. Parquet is growing in popularity as a format in the big data world as it allows for faster query run time, it is smaller in size and requires fewer data to be scanned compared to formats such…