A Data Engineering Portal
In this blog, you will be learning different ways to remove data from a table using SQL. These statements might not work for all the databases. What is the DROP Command in SQL? The SQL DROP command is a DDL (Data Definition Language) command that deletes the defined table with all its table data, associated […]
In this blog, You will learn about the Databrick cluster access modes Cluster Access Modes There are 4 types of access modes for the cluster Single user Multiple user access No Isolation shared Custom access user You can watch out my youtube tutorial on Databricks
As we all know, Data is the backbone of any system. This data in the form of bits need to be stored for required processing in real-time or needs a basis What is Data Storage? Data storage is the recording & storing of data/information in a storage medium. Data storage could be in the form […]
To start any Spark application on a local Cluster/yarn, we need to set some configuration and parameters, and it can be done using SparkConf. Below are the basic parameter required: These properties can be set directly on a SparkConf & passed to SparkContext. Let us consider the following example of using SparkConf in a PySpark […]
In this blog, You will learn how to map other columns of the dataframe in PySpark. Let’s understand the data first. You can see the ID, Name, and Age columns here. Now, the requirement is to create another column that will have the mapping of Name and Age columns (i.e {Name->’Sagar’, Age->25}). Let’s understand how […]
In this blog, You will be solving Zomato SQL Data Analysis Project. You will be given a dummy dataset having a total of four tables that is similar to zomato company data. This project is for those candidates who are very beginners in this field. This project is totally on SQL, You will not be […]
In this blog, You will learn what is the advantage of creating a broadcast variable and how to use it in the dataframe using PySpark using Databricks.For parallel processing, Apache Spark uses shared variables. A copy of the shared variable goes on each node of the cluster when the driver sends a task to the […]
In this blog, you will learn about the music store data analysis project. If you are a beginner or have some basic knowledge of Data Analysis then this is gonna be a very exciting project for you. Let’s understand the questions first. Questions to be solved Who is the senior most employee based on job […]
In this blog, You will learn how to find out the latest folder from the file path in the databricks. Here the file source could be anything like ADLS Gen2, GEN1, Blob Storage, DBFS, and S3 bucket. Assuming, data stored in the source file path is saved in date partitioned. And now, you need to […]
In this blog, You will learn how to build a chatbot using chatGPT and python. You must have tried chatGPT from the portal but today you will how to make the same thing with the help of python language. Since you need the power of chatGPT so you can easily connect to chatGPT via API. […]
Price Based Country test mode enabled for testing Australia. You should do tests on private browsing mode. Browse in private with Firefox, Chrome and Safari
