Wednesday, July 31, 2024

Python in 30 min

 Python is a versatile and powerful programming language that's widely used in various fields, including web development, data analysis, artificial intelligence, and more. Here’s a beginner-friendly tutorial to get you started with Python:

1. Introduction to Python

Python is known for its readability and simplicity. It uses indentation to define code blocks, which makes it visually clear and easy to follow.

2. Setting Up Python

  1. Download and Install Python:

    • Go to the official Python website and download the latest version.
    • Follow the installation instructions for your operating system (Windows, macOS, or Linux).
  2. Install an IDE or Text Editor:

    • You can write Python code in various IDEs and text editors. Some popular choices are:
      • IDLE: Comes bundled with Python.
      • PyCharm: A powerful IDE for Python.
      • VS Code: A lightweight but powerful editor with Python support.

3. Basic Syntax and Concepts

Hello World

Let's start with a simple program that prints "Hello, World!" to the console.


print("Hello, World!")

Variables and Data Types

Variables are used to store data. Python supports various data types including integers, floats, strings, and booleans.


x = 5 # Integer y = 3.14 # Float name = "Alice" # String is_student = True # Boolean

Basic Operations


# Arithmetic operations a = 10 b = 5 print(a + b) # Addition print(a - b) # Subtraction print(a * b) # Multiplication print(a / b) # Division # String concatenation first_name = "John" last_name = "Doe" full_name = first_name + " " + last_name print(full_name)

Control Structures

Conditional Statements


age = 18 if age >= 18: print("You are an adult.") else: print("You are a minor.")

Loops

  • For Loop

for i in range(5): print(i)
  • While Loop

count = 0 while count < 5: print(count) count += 1

4. Functions

Functions are reusable blocks of code that perform a specific task.


def greet(name): return f"Hello, {name}!" print(greet("Alice"))

5. Lists and Dictionaries

  • Lists are ordered collections of items.

fruits = ["apple", "banana", "cherry"] print(fruits[0]) # Access first item fruits.append("date") # Add item
  • Dictionaries are collections of key-value pairs.

person = {"name": "John", "age": 30} print(person["name"]) # Access value by key person["age"] = 31 # Update value

6. File Handling

You can read from and write to files using Python.

Reading a file


with open('example.txt', 'r') as file: content = file.read() print(content)

Writing to a file


with open('example.txt', 'w') as file: file.write("Hello, World!")

7. Modules and Packages

Modules and packages allow you to organize your code into separate files and directories.

Importing a module


import math print(math.sqrt(16))

Creating your own module Save the following in a file named my_module.py:


def say_hello(name): return f"Hello, {name}!"

You can use it in another file:


import my_module print(my_module.say_hello("Alice"))

8. Error Handling

Python uses try and except blocks to handle errors gracefully.


try: result = 10 / 0 except ZeroDivisionError: print("You can't divide by zero!") finally: print("This block always executes.")

9. Object-Oriented Programming

Python supports object-oriented programming. You can create classes and objects.




class Dog: def __init__(self, name): self.name = name def bark(self): return f"{self.name} says Woof!" my_dog = Dog("Buddy") print(my_dog.bark())

10. Next Steps

  • Explore Libraries: Python has a rich ecosystem of libraries and frameworks. Explore libraries like NumPy for numerical computing, pandas for data analysis, and Flask/Django for web development.
  • Practice: Work on small projects or problems to reinforce your learning.

Feel free to ask if you have questions about any of these topics or need more detailed explanations!

Tuesday, July 23, 2024

Databricks - File to Table Data Loading


# Step-by-Step Script for File to Table Data Loading in Databricks:

from pyspark.sql import SparkSession


# Initialize Spark session

spark = SparkSession.builder \

                    .appName("File to Table Data Loading") \

                    .getOrCreate()


# Load data from CSV files into DataFrame

df = spark.read.format("csv") \

               .option("header", "true") \

               .load("dbfs:/mnt/data/csv_files/")


# Perform data transformations if needed

df = df.withColumn("amount", df["amount"].cast("double"))


# Save DataFrame to a Delta Lake table

df.write.format("delta") \

        .mode("overwrite") \  # or "append" for incremental loading

        .saveAsTable("my_database.my_table")


# Optionally, stop Spark session

spark.stop() 

Azure Data Factory Azure Data Factory - list of activities

List of Activities in ADF Azure Data Factory Azure Data Factory is a cloud-based data integration service that allows you to create, schedu...