Some architectural examples of how we can process data in AWS depending on our data needs. — There are many ways to process data in AWS with the many AWS Services that are available. There is also a growing demand in organizations to update systems in more real time. In my opinion, Data engineers should not silo themselves by just having skills to build batch processing data…

So when you create a brand new aws glue job, I don’t know about you but it seems pretty intimidating that there are 6 python import statements that are generated automatically. I wanted to understand why we needed them so I did a bit of research and made this post…

There are a lot more people that know SQL than know how to program in python and are proficient in Spark to perform big data analytics on their data. With AWS Glue Studio, it’s possible to build data pipelines for big data analytics on a distributed cluster without knowing to…

How to create, insert, and query records in your governed table using the AWS Data Wrangler library in Python — Governed Tables, is a new type of table on Amazon S3 that supports ACID transactions which can help you build more resilient data pipelines for your data lake on AWS. Governed tables can be enabled on top of your data in AWS S3 in the following formats: Avro, CSV, JSON…

When it comes to data analytics and building data pipelines in python, you have probably heard of pandas which is versatile for data analytics on a single machine and Pyspark for data analytics in a distributed environment when you are working with too much data for one machine to handle…

The video below provides an overview of AWS Glue Studio which is part of the AWS Glue Service. This is a step-by-step walkthrough of the various components of AWS Glue Studio and how to use use it.

Pandas is an extremely popular and essential python package for data science as it’s powerful, flexible and easy to use open-source data analysis and data manipulation. …

Parquet is an open-sourced columnar storage format created by the Apache software foundation. Parquet is growing in popularity as a format in the big data world as it allows for faster query run time, it is smaller in size and requires fewer data to be scanned compared to formats such…

On July 9th, 2020, Safe Software Inc. released FME 2020.1. Here are the top features I’m excited about. I have to say this is a pretty significant .1 release comparted to some .1 releases in previous years. Readers and Writers: FME has always had unrivalled spatial support and they have continued to expand…