Data World Solution

https://youtu.be/gHp7Gc7sXMk

In this tutorial, we explore the final method of loading data into Microsoft Fabric Warehouse — Cross Database Ingestion. Learn how to use CTAS, INSERT INTO, and SELECT INTO to copy data from one database to another. Perfect for those working with multiple databases in Fabric Warehouse.

1 month ago | [YT] | 0

Data World Solution

https://youtu.be/mjfTu6KIGO8

2 months ago | [YT] | 0

Data World Solution

In Microsoft Fabric, the ForEach activity allows looping over a collection (like a list of items) in a Data Factory pipeline. You can use it to perform actions on each item in the collection. For example, after using the Get Metadata activity to retrieve a list of files in a folder, ForEach can iterate through each file. Within each iteration, you can add activities like setting a variable to store each file's name. This loop structure executes the activities for each item in the list, enabling flexible and repeated processing within a pipeline.


For Complete Demo Watch Video -
--------------------------------------------------------------------------------------------------------------------------
#MicrosoftFabric #DataFactory #ForEachActivity #DataPipelines #Automation #Azure #DataEngineering #CloudComputing #TechTutorial #DataTransformation

10 months ago | [YT] | 2

Data World Solution

Ready to level up your data processing skills in Azure Databricks? Learn how to read files into Spark DataFrames and define custom schemas for efficient data management. Please watch this video to learn.

Code used in video :-

display(dbutils.fs.mounts())
%fs
ls /mnt/dws4dl4gen2/demo
countries_df= spark.read.csv("/mnt/dws4dl4gen2/demo/countries.csv")
type(countries_df)
countries_df.show()
display(countries_df)
countries_df.display()
countries_df= spark.read.csv("/mnt/dws4dl4gen2/demo/countries.csv",header = True)
countries_df.display()
countries_df= spark.read.options(header = True ).csv("/mnt/dws4dl4gen2/demo/countries.csv")
countries_df.display()
countries_df.dtypes
countries_df.printSchema()
countries_df.describe()
countries_df= spark.read.options(header = True , inferSchema = True).csv("/mnt/dws4dl4gen2/demo/countries.csv")
countries_df.dtypes
from pyspark.sql.types import IntegerType, StringType, DoubleType, StructField, StructType
countries_schema = StructType([
StructField("COUNTRY_ID", IntegerType(), False),
StructField("NAME", StringType(), False),
StructField("NATIONALITY", StringType(), False),
StructField("COUNTRY_CODE", StringType(), False),
StructField("ISO_ALPHA2", StringType(), False),
StructField("CAPITAL", StringType(), False),
StructField("POPULATION", DoubleType(), False),
StructField("AREA_KM2", IntegerType(), False),
StructField("REGION_ID", IntegerType(), True),
StructField("SUB_REGION_ID", IntegerType(), True),
StructField("INTERMEDIATE_REGION_ID", IntegerType(), True),
StructField("ORGANIZATION_REGION_ID", IntegerType(), True)
]
)
countries_df= spark.read.csv("/mnt/dws4dl4gen2/demo/countries.csv",header= True , schema=countries_schema)
countries_df.dtypes
countries_df= spark.read.options(header= True ).schema(countries_schema).csv("/mnt/dws4dl4gen2/demo/countries.csv")
countries_df.dtypes
countries_df.printSchema()

11 months ago | [YT] | 1

Data World Solution

Magic Commands Purpose: Allow switching between languages in a Databricks notebook and perform various tasks.
Language Switching: Use %SQL to run SQL, %Scala to run Scala, and %md for markdown documentation.
Execution: Commands like %SQL and %Scala change the cell’s context. You can run the entire notebook using the "Run all" button, which executes cells in their respective languages.
Markdown Documentation: %md enables documentation with markdown syntax, including headers, bullet points, and HTML embedding.
File System Command: %fs allows interaction with the file system, such as listing files.
Shell Command: %sh runs shell commands, like ps to view running processes.
Utility: Magic commands enable the use of multiple languages and utilities within a single notebook, enhancing flexibility and functionality.

11 months ago | [YT] | 2

Data World Solution

Databricks offers enhanced Jupyter-style notebooks that execute commands on a cluster, with features for organizing, running, and sharing work. Notebooks support multiple languages through Magic Commands and must be attached to a cluster for execution. Users can run cells individually or all at once, monitor execution time, and utilize keyboard shortcuts for efficiency. Notebooks can be shared, scheduled via Databricks Jobs, and exported in various formats.

Watch Video for Demo⤵️▶️

11 months ago | [YT] | 1

Data World Solution

A Cluster is basically a collection of virtual machines.
In a cluster, there is usually a driver node, which orchestrates and the tasks performed by one or more worker nodes.

Cluster allow us to treat this group of computers, as single compute engine via the driver node.

Cluster Types

Single Node Cluster: Only one node (Driver Node). No Worker Nodes. Supports Spark workloads. Suitable for lightweight Machine Learning and Data Analysis. Not horizontally scalable.
Multi Node Cluster: One Driver Node and one or more Worker Nodes. Horizontally scalable. Suitable for large workloads like Spark Jobs.
Access Modes

Single User: Only one user access. Supports Python, SQL, Scala, and R.
Shared: Multiple users, process isolation. Available on premium workspaces. Supports Python and SQL.
No Isolation Shared: Multiple users, no process isolation. Supports all four languages. Less secure.
Custom: Legacy option, not available in the latest interface.
Databricks Runtimes

Databricks Runtime: Includes Apache Spark, libraries for Java, Scala, Python, R, and GPU.
Databricks Runtime ML: Adds machine learning libraries like PyTorch, TensorFlow.
Photon Runtime: Adds Photon Engine for faster SQL workload processing.
Databricks Runtime Light: For automated workloads, no advanced features.
Auto Termination

Terminates idle clusters to avoid costs. Default value is 120 minutes, adjustable between 10-10,000 minutes.
Auto Scaling

Dynamically adjusts the number of Worker Nodes based on workload. Not recommended for streaming workloads.
Azure VM Types

Memory Optimized: For memory-intensive tasks like ML.
Compute Optimized: For structured streaming applications.
Storage Optimized: For high disk throughput needs.
General Purpose: For enterprise-grade applications and analytics.
GPU Accelerated: For deep learning models.
Cluster Policies

Simplifies cluster configuration for standard users. Limits cluster size and options to ensure cost control. Available only on premium tier.

11 months ago | [YT] | 1

Data World Solution

https://youtu.be/gQ1YVLrYdkw?si=ZkHM2...

Skip the Carousel

1 2 3 4

1 year ago | [YT] | 7

Data World Solution

Learn how to efficiently manage column truncation using the SIZE_LIMIT, RETURN_FAILED_ONLY, TRUNCATECOLUMNS, FORCE option in Snowflake's COPY command!
#Snowflake #DataLoading #DataManagement #SQL #COPYINTOCOMMAND

1 year ago | [YT] | 1

Data World Solution

Learn how to effortlessly rename columns and tables in Snowflake for streamlined data management and organization! Watch now!

1 year ago | [YT] | 1