PySpark Online Compiler: Write, Run, and Test Code Effectively

In this PySpark Playground, you have access to several sample datasets stored in the /samples/ folder. These datasets can be used to test your PySpark code and understand how to work with real-world data.

Example: `customers.csv`

The customers.csv file is a sample dataset that contains customer information. You can load this file into a DataFrame using PySpark and apply various transformations and actions on it. Below is a code snippet to help you get started:

# Initialize Spark session
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Spark Playground').getOrCreate()

# Load the customers.csv dataset
df = spark.read.format('csv').option('header', 'true').load('/samples/customers.csv')

# Show the first few rows of the DataFrame
df.show(5)

# Display the DataFrame using the display() function.
display(df)

Available Sample Datasets:

customers.csv: Contains customer data with fields like customer ID, name, and purchase history.
sales.csv: Records of sales transactions with fields like transaction ID, date, and amount.
products.csv: Information about products, including product ID, name, and price.

Try it out!

Use the provided datasets to explore various PySpark functions such as select(), filter(), groupBy(), and more. This will help you get hands-on practice with data processing in PySpark.

Environment and Library versions:

Python : 3.8.20
PySpark : 3.5.3
Pandas : 2.0.3

TABLE STDOUT STDERR

Use display(df) function to show the final DataFrame in this section.