An introduction to an effective logging approach — When working with any programming language, the end goal is always to have a running program that functions according to the provided instructions. However, it is rare to achieve the desired outcome on the first try and you may need to analyze your runs to identify errors. It has become…

Learning how to load variables securely to your code — When you push your code to a GitHub repository or use it in production systems, there may be a need to load certain secure variables. For instance, API keys that you do not want to be visible publicly. When working on production with CI/CD systems, you may have to change…

A simple introduction to memory-effective iteration in Python — This article focuses on one of the essential concepts of data processing in Python, which is iteration. In simple words, iteration helps us overcome the problem of limited memory by allowing us to use the data one at a time, as and when needed, thereby enabling lazy execution. …

Everything you need to communicate with the Kafka cluster — Introduction Learning to communicate with a new tool is essential, and in this article, we will get hands-on experience working with Kafka. To be able to give instructions to Kafka, we need to know some of the Kafka CLI commands, which are commonly used. Although we use Java as the primary…

Fundamentals to gain an understanding of Kafka's architecture — Introduction Modern data engineering workflows are increasingly shifting from traditional batch processing to real-time data processing. This option is particularly useful when you require real-time control over some process. The first step in this process is to extract data and load it into storage or use other outputs. The architecture can…

Important to know before optimizing query performance — To maximize your query’s speed on any SQL engine, it’s essential to have an understanding of the SQL execution order. Even though you can work without this knowledge, I recommend reading this article to gain a quick understanding of it. The SQL engine doesn’t follow the same order as you…

Learn essential Git commands — committing, branching, and merging — What is Git? In most of the sources, you will find relatively short and self-explaining definitions of Git — a version control system (VCS). But what exactly is a version control system? Basically, this system helps us track and manage changes to files over time. When working in groups, we deal with…

Introduction to Computer Science Concepts — Many aspiring developers tend to jump straight into learning programming languages without getting a good grasp of the fundamental concepts of Computer Science. However, as they progress in their careers, they eventually realize the importance of understanding how a computer actually operates. …

A Practical Guide to AWS Glue — Introduction You may notice that the S3 bucket contains raw data files after data extraction, which can be in various formats such as CSV, JSON, and Parquet. To use this data for analytical processes, it needs to be transformed and loaded into a database or data warehouse. …