Refactor code structure for improved readability and maintainability

2026-07-22 22:58:28 +05:30
parent d1308bf149
commit bc8bf2007d
99 changed files with 4784 additions and 111 deletions
@@ -0,0 +1,69 @@
+# CRM Chat + Translate Pod (self-bootstrapping)
+
+One RunPod **Pod** running **Qwen2.5-14B** (chat/assistant, GPU) + **NLLB-200-1.3B**
+(translation, CPU). It **auto-starts on every boot/resume** — you only Stop/Resume;
+the CRM team's URLs never change and nothing needs editing.
+
+## One-time: create the Pod
+RunPod → **Pods → Deploy**:
+
+| Setting | Value |
+|---|---|
+| **GPU** | **RTX 3090 (24 GB)** — On-Demand / Secure Cloud (not Spot) |
+| **Template** | RunPod PyTorch (has python3 + CUDA) |
+| **Network Volume** | create one, **~50 GB**, mount at **`/workspace`** ← holds models + deps so Stop/Resume keeps them |
+| **Container Disk** | **40 GB** |
+| **Expose HTTP Ports** (Edit Template) | `8000,11434` |
+| **Docker / Start Command** | see below |
+
+**Start Command** (paste exactly — this is the whole setup):
+```
+bash -c "curl -fsSL https://raw.githubusercontent.com/Sumit-Pluto/photo_gallery/main/chat-pod/start.sh | bash"
+```
+
+First boot downloads Qwen (~16 GB) + converts NLLB → **~15–20 min**. Every Resume
+after that: models already on the volume → **ready in ~1–2 min**, same URLs.
+
+## The two URLs to share with the CRM team
+After deploy, each exposed port has a stable proxy URL (stable as long as you
+**Stop/Resume**, never Terminate):
+```
+Chat  (Qwen, OpenAI-compatible):  https://<POD_ID>-11434.proxy.runpod.net/v1/chat/completions
+Translate (NLLB):                 https://<POD_ID>-8000.proxy.runpod.net/translate
+```
+
+## API for the CRM
+**Chat / summarize** (OpenAI format, supports `stream:true`):
+```json
+POST /v1/chat/completions
+{ "model": "qwen2.5:14b-instruct-q8_0", "stream": true,
+  "messages": [{ "role": "user", "content": "Summarize this thread in English: ..." }] }
+```
+**Translate before send / on receive:**
+```json
+POST /translate
+{ "text": "When can you deliver the cement?", "source": "eng_Latn", "target": "hin_Deva" }
+-> { "translation": "..." }
+```
+
+## FLORES language codes (NLLB)
+`eng_Latn` English · `hin_Deva` Hindi · `spa_Latn` Spanish · `fra_Latn` French ·
+`deu_Latn` German · `arb_Arab` Arabic · `zho_Hans` Chinese · `rus_Cyrl` Russian ·
+`por_Latn` Portuguese · `ben_Beng` Bengali · `tam_Taml` Tamil · `tel_Telu` Telugu ·
+`mar_Deva` Marathi · `guj_Gujr` Gujarati · `pan_Guru` Punjabi · `jpn_Jpan` Japanese ·
+`kor_Hang` Korean · `vie_Latn` Vietnamese · `ind_Latn` Indonesian
+(full list: NLLB-200 FLORES-200 codes.) Your CRM maps each user's language → its code.
+
+## Security (do this)
+- **Only call these URLs from your CRM backend**, never the browser (the proxy URLs are public).
+- Optional: set env **`CHAT_POD_KEY`** on the pod → the translate API then requires
+  `Authorization: Bearer <key>`. (Ollama has no built-in auth — keep it backend-only or front it with a gateway.)
+
+## Daily operation (what you actually do)
+- **Start working:** Pod → **Resume** → wait ~1–2 min → team uses the same URLs.
+- **Done:** Pod → **Stop** (stops GPU billing; keeps the volume + URL).
+- **Never Terminate** — that deletes the pod and changes the URL.
+
+## Tuning
+- Lighter/faster model: set env `QWEN_MODEL=qwen2.5:14b` (Q4, ~9 GB) or `qwen2.5:7b`.
+- The script reads `QWEN_MODEL`, `NLLB_HF`, `CHAT_POD_KEY` from env if you want to override.
@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# =============================================================================
+# CRM chat + translate pod — self-bootstrapping start script.
+#
+# Set this as the Pod's Docker/Start Command (one line):
+#   bash -c "curl -fsSL https://raw.githubusercontent.com/Sumit-Pluto/photo_gallery/main/chat-pod/start.sh | bash"
+#
+# It is IDEMPOTENT: safe to run on first boot AND on every Resume. Everything
+# heavy (models + python deps) lives on the /workspace VOLUME, so Stop/Resume
+# keeps it — the CRM team just needs the two proxy URLs, nothing to edit.
+#
+#   Qwen2.5-14B (chat/assistant)  -> GPU, Ollama,  port 11434 (OpenAI-compatible)
+#   NLLB-200-1.3B (translation)   -> CPU, CTranslate2 + FastAPI, port 8000
+# =============================================================================
+set -uo pipefail
+
+WORK=/workspace
+VENV="$WORK/venv"
+export OLLAMA_MODELS="$WORK/ollama"
+export OLLAMA_HOST=0.0.0.0
+export NLLB_HF="facebook/nllb-200-1.3B"
+export NLLB_CT2="$WORK/nllb-ct2"
+QWEN_MODEL="${QWEN_MODEL:-qwen2.5:14b-instruct-q8_0}"   # 8-bit ~16GB; override via env
+
+mkdir -p "$WORK" "$OLLAMA_MODELS"
+echo "[chat-pod] boot — models/deps on $WORK (persist across Stop/Resume)"
+
+# --- 0. system deps: lspci lets Ollama auto-detect the GPU (else it may use CPU) ---
+if ! command -v lspci >/dev/null 2>&1; then
+  echo "[chat-pod] installing pciutils (GPU detection)..."
+  apt-get update -qq >/dev/null 2>&1 && apt-get install -y -qq pciutils >/dev/null 2>&1 || true
+fi
+
+# --- 1. Ollama (installs to container disk; reinstalled cheaply if missing) ----
+if ! command -v ollama >/dev/null 2>&1; then
+  echo "[chat-pod] installing Ollama..."
+  curl -fsSL https://ollama.com/install.sh | sh
+fi
+
+# --- 2. Python deps on the VOLUME (persist) -----------------------------------
+if [ ! -x "$VENV/bin/uvicorn" ]; then
+  echo "[chat-pod] creating venv + installing NLLB deps (one time)..."
+  python3 -m venv "$VENV"
+  "$VENV/bin/pip" install -q --upgrade pip
+  "$VENV/bin/pip" install -q ctranslate2 transformers sentencepiece fastapi uvicorn
+fi
+# torch (CPU) is required to CONVERT the NLLB model; ensure it exists on any venv.
+"$VENV/bin/python" -c "import torch" >/dev/null 2>&1 || {
+  echo "[chat-pod] installing CPU torch (needed to convert NLLB)..."
+  "$VENV/bin/pip" install -q torch --index-url https://download.pytorch.org/whl/cpu
+}
+
+# --- 3. Fetch the translate server (always latest from repo) -------------------
+curl -fsSL https://raw.githubusercontent.com/Sumit-Pluto/photo_gallery/main/chat-pod/translate.py -o "$WORK/translate.py" \
+  || echo "[chat-pod] WARN: could not refresh translate.py (using existing)"
+
+# --- 4. Start Ollama, wait for it, ensure Qwen is pulled (idempotent) ---------
+echo "[chat-pod] starting Ollama (GPU)..."
+ollama serve &
+for i in $(seq 1 30); do curl -s http://localhost:11434/api/tags >/dev/null 2>&1 && break; sleep 2; done
+echo "[chat-pod] ensuring model $QWEN_MODEL (first time downloads ~16GB)..."
+ollama pull "$QWEN_MODEL" || echo "[chat-pod] WARN: pull failed (will use cached if present)"
+
+# --- 5. Convert NLLB once (cached on the volume after) ------------------------
+if [ ! -f "$NLLB_CT2/model.bin" ]; then
+  echo "[chat-pod] converting NLLB -> CTranslate2 int8 (one time)..."
+  rm -rf "$NLLB_CT2"   # clear any partial/failed conversion
+  "$VENV/bin/ct2-transformers-converter" --model "$NLLB_HF" --output_dir "$NLLB_CT2" --quantization int8
+fi
+
+# --- 6. Start the NLLB translate API (CPU) -----------------------------------
+echo "[chat-pod] starting NLLB translate API (CPU) on :8000..."
+cd "$WORK"
+"$VENV/bin/uvicorn" translate:app --host 0.0.0.0 --port 8000 &
+
+echo "[chat-pod] READY  ->  Qwen :11434 (chat)  |  NLLB :8000 (translate)"
+wait
@@ -0,0 +1,56 @@
+"""
+NLLB-200 translation API (CPU, CTranslate2 int8). Part of the CRM chat pod.
+Runs on port 8000 alongside Ollama/Qwen (port 11434) on the same pod.
+
+POST /translate  {"text": "...", "source": "eng_Latn", "target": "hin_Deva"}
+              -> {"translation": "..."}
+GET  /health  -> {"ok": true}
+
+Optional auth: set env CHAT_POD_KEY, then callers must send
+`Authorization: Bearer <key>`.
+"""
+
+import os
+
+import ctranslate2
+import transformers
+from fastapi import FastAPI, Header, HTTPException
+from pydantic import BaseModel
+
+CT2 = os.environ.get("NLLB_CT2", "/workspace/nllb-ct2")
+HF = os.environ.get("NLLB_HF", "facebook/nllb-200-1.3B")
+API_KEY = os.environ.get("CHAT_POD_KEY", "")
+
+_translator = ctranslate2.Translator(CT2, device="cpu", compute_type="int8")
+_tok = transformers.AutoTokenizer.from_pretrained(HF)
+
+app = FastAPI(title="CRM Translate (NLLB-200)")
+
+
+class Req(BaseModel):
+    text: str
+    source: str  # FLORES code, e.g. "eng_Latn"
+    target: str  # FLORES code, e.g. "hin_Deva"
+
+
+def _check(authorization: str) -> None:
+    if API_KEY and authorization != f"Bearer {API_KEY}":
+        raise HTTPException(status_code=401, detail="unauthorized")
+
+
+@app.post("/translate")
+def translate(r: Req, authorization: str = Header(default="")):
+    _check(authorization)
+    text = (r.text or "").strip()
+    if not text:
+        return {"translation": ""}
+    _tok.src_lang = r.source
+    source = _tok.convert_ids_to_tokens(_tok.encode(text))
+    results = _translator.translate_batch([source], target_prefix=[[r.target]])
+    hyp = results[0].hypotheses[0][1:]  # drop the leading target-language token
+    return {"translation": _tok.decode(_tok.convert_tokens_to_ids(hyp))}
+
+
+@app.get("/health")
+def health():
+    return {"ok": True}