Data-Analytics/Day_002_Core_Data_Processing.md

Day 2 — Core Data Processing with Python

File: Day2_Core_Data_Processing.md

# 📅 Day 2 — Core Data Processing with Python

## 🎯 Goal
Transform raw data into structured, insightful information using Python.

---

## 🧩 Tasks

### 1. Integrate Python in n8n
- Add an **Execute Code** node after the Merge node (from Day 1).  
- Receive combined JSON data from sales and reviews.

---

### 2. Data Cleaning & Aggregation
- Load JSON data into two Pandas DataFrames:
  ```python
  import pandas as pd

  sales_df = pd.DataFrame($json["sales"])
  reviews_df = pd.DataFrame($json["reviews"])
Clean data: handle missing values, convert data types, remove duplicates.

Aggregate sales data:

python
Copy code
sales_summary = sales_df.groupby("product_id").agg(
    total_revenue=("price", "sum"),
    units_sold=("quantity", "sum")
).reset_index()
3. Sentiment Analysis
Use NLTK VADER for review sentiment:

python
Copy code
from nltk.sentiment.vader import SentimentIntensityAnalyzer
sid = SentimentIntensityAnalyzer()

reviews_df["sentiment_score"] = reviews_df["review_text"].apply(
    lambda text: sid.polarity_scores(text)["compound"]
)
Categorize sentiment: Positive, Neutral, Negative.

Aggregate sentiment per product:

python
Copy code
sentiment_summary = reviews_df.groupby("product_id").agg(
    avg_sentiment_score=("sentiment_score", "mean"),
    num_reviews=("review_text", "count")
).reset_index()
4. Combine & Output
Merge aggregated sales and sentiment:

python
Copy code
final_df = pd.merge(sales_summary, sentiment_summary, on="product_id", how="left")
return json.loads(final_df.to_json(orient="records"))
✅ Deliverable
Python node outputs a clean JSON object containing:

Aggregated sales data

Sentiment scores per product

💡 Solution
Combined DataFrame ready for storage and reporting in Day 3.

All data is clean, structured, and enriched.