AI / ML Engineer · Computer Vision · Generative AI

Rohit
Paul.

I build production-grade AI systems — real-time lip-sync pipelines, diffusion models, and distributed training infrastructure. Currently shipping at Alethia AI.

View My Work Get in Touch
Rohit Paul
Open to RolesAI/ML · CV · GenAI
SCROLL
<1s
First-frame latencyMoDA lip-sync pipeline
4+
YearsAI/ML experience
IEEE
INDICON 2021Published research
9.03
MTech GPAIIEST Shibpur
3
CompaniesShipped production AI
115
HackerRankPython challenges ✓
10×
Speed-upTensorRT optimised
<1s
First-frame latencyMoDA lip-sync pipeline
4+
YearsAI/ML experience
IEEE
INDICON 2021Published research
9.03
MTech GPAIIEST Shibpur
3
CompaniesShipped production AI
115
HackerRankPython challenges ✓
10×
Speed-upTensorRT optimised
Experience

Where I've
Made an Impact

From published research to production AI — building systems that actually ship.

Feb 2020 — May 2021
MTech Thesis
VarDenGrid — Variable Density Clustering
Published research · IEEE INDICON 2021 · IIEST Shibpur
  • Designed a novel clustering algorithm with grid-based initialisation that handles variable-density clusters — overcoming DBSCAN's single-epsilon limitation
  • Requires zero user input; parameter estimation is fully automatic across datasets
  • Benchmarked against DBSCAN and OPTICS — comparable performance on standard benchmarks
  • Accepted and published at IEEE INDICON 2021 — International Conference of the IEEE India Council
ClusteringPython Scikit-learnIEEE Published
Oct 2024 — Present
Alethia AI
AI/ML Computer Vision Engineer
Real-time lip-sync · emotive avatars · generative pipelines
  • Optimised real-time lip-sync pipeline (MoDA) — reduced first-frame latency from 10s to under 1s via TensorRT, Kalman smoothing, and progressive batching
  • Resolved visual artifacts (flickering, wandering eyes, pursed lips) with automated mouth-opening correction for diverse faces
  • Built emotive lip-sync with emotion-conditional CFG, intra-emotion interpolation, and Cartesia SSML audio
  • Trained JoyVASA+LivePortrait on MEAD/HDTF; designed fault-tolerant distributed training (DDP, Ray Tune, S3 auto-upload) on VastAI/IONet
  • Deployed Gemini+Flux image generation APIs with multi-key quota rotation and SQS health checks on GCP/Azure; Flux pipeline on RunPod
  • Benchmarked LatentSync, HunyuanVideo-Avatar, Multi-Talk, WanS2V, FramePack with cost-per-second analysis
  • Systematically benchmarked SOTA image/video generation models with advanced prompt templating and fault tolerance to automatically recover from erroneous user prompts
  • Automated a meme generation pipeline integrating image generation models with Google OCR to autonomously detect and correct distorted text
  • Developed a comprehensive web UI scraper for curating high-quality lip-sync data from YouTube playlists, utilizing automatic cropping and VLM/LLM filtration for strict mouth visibility and single-character criteria
MoDATensorRTDDP Ray TuneFluxGCP · Azure RunPodPrompt EngOCRData Scraping
Jun 2023 — Oct 2024
DressMe AI
AI/ML Engineer — Generative AI & CV
Virtual try-on · diffusion models · distributed training
  • Developed and deployed diffusion-based virtual try-on models; fine-tuned PHNet image harmonisation model
  • Built multi-GPU distributed training pipelines for large-scale dataset preparation and model training
  • Implemented image relighting via Three.js virtual light source
  • Fine-tuned PHNet image harmonization model by leveraging custom datasets generated via Three.js data augmentation
Diffusion ModelsPHNet Multi-GPUThree.js Data Augmentation
Personal Project
ClawdBot
AI Engineer — Agentic Systems & Real-Time Video
Agentic workflows · WebGL lip-sync · video pipelines
  • Built agentic AI workflows using LangChain for automated daily AI news updates with context
  • Developed WebGL-based real-time lip-sync system and video creation pipelines using Remotion
LangChainWebGL RemotionAgentic AI
Skills

Technical Arsenal

Tools I use to build, train, optimise, and deploy AI systems at scale.

🧠ML / AI Frameworks
PyTorchTensorFlowHuggingFaceDiffusersSklearnLangChainGradioRAG
⚡Inference & Optimisation
TensorRTONNXfp16DDPRay Tune
👁Vision & Media
OpenCVFFmpegWebGLThree.jsComfyUI
☁️Cloud & Deployment
AWS S3/SQS/EC2GCPAzureRunPodVastAIReplicatefal.aiDocker
💻Languages
PythonC++JavaScriptJavaSQL
🛠Dev Tools
GitLinuxFastAPIFlaskVSCodeCursorClawdBot
Projects

Everything I've
Built & Shipped

22 projects across Alethia AI, DressMe AI, and personal work. Click any card to see details.

Alethia AI⚡
MoDA Real-Time Lip-Sync Pipeline
Reduced first-frame latency from 10s to <1s via TensorRT, Kalman smoothing, and progressive batching.
Aug–Oct 2025→
Alethia AI😊
Emotive Lip-Sync System
Emotion-conditional CFG, intra-emotion interpolation, emotion transfer, and Cartesia SSML audio integration.
Sep–Nov 2025→
Alethia AI🔧
Lip-Sync Artifact Resolution
Systematic fixes: mumbling, flickering, wandering eyes, pursed lips — including an automated mathematical mouth-opening solution.
Oct–Dec 2025→
Alethia AI🤖
JoyVASA + LivePortrait Training
Merged JoyVASA and LivePortrait. Manual controls for head rotation, eye gaze, mouth shapes. 6D rotation + Full UpsampleConformerEncoder.
Jan–Jul 2025→
Alethia AI🏗
Distributed Training Infrastructure
Fault-tolerant training across VastAI and IONet. Ray Tune + DDP + TensorBoard. S3 auto-upload with resume-from-checkpoint.
Jun–Jul 2025→
Alethia AI🗂
Dataset Preparation Pipeline
Web app: input YouTube playlists → scrapes and clips them into training data. CV + VLM/LLM filtration. 20 hours of clean video produced.
Jul–Aug 2025→
Alethia AI🖼
Image Generation API — Gemini + Flux
Multi-key Gemini quota rotation with Flux fallback. CPU deployment on GCP/Azure. SQS health checks. Flux avatar pipeline on RunPod.
Mar–May 2025→
Alethia AI📊
Model Benchmarking & Analysis
Cost-per-second analysis across LatentSync, HunyuanVideo-Avatar, Multi-Talk, WanS2V, FramePack, and background removal models.
Mar–Nov 2025→
Alethia AI📡
Streaming Protocol Analysis
OME with SRT vs WebRTC vs HLS deep-dive. PersonaLive vs LivePortrait evaluation. Established 2026 MoDA scaling plan.
Dec 2025→
Alethia AI🎨
Avatar Style & Image Pipelines
Fixed Pixar/anime/3D style outputs. Flux inpaint pipeline. Identity-preservation model for professional selfie generation with Seedream.
Jan–Feb 2025→
Alethia AI🎬
Generative Scene & Asset Pipeline
UNO and Flux Kontext for identity-preserved scenes. LTX-Video pipeline with Gemini filtration. KEEP face upscaling.
May 2025→
Alethia AI🎭
Emotional Facial Expression Gen
LLM-driven frame-wise expression changes from audio. Smile, sad, angry, woo, wink, blink transitions. fp16 optimised.
Mar 2025→
Alethia AI🖥
Gradio UI for Marketing Team
Integrated Ditto + LatentSync into an internal tool. Smile dampening, silent lip adjustment. Unblocked daily AINS video production.
Mar 2025→
Alethia AI🧩
SCAIL + 2D Character Generation
Automated 2D character generation and emotive base video creation using SCAIL. Standardised consistency across states.
Dec 2025→
Alethia AI📺
Morgan Livestream Preparation
Prepared base assets with varied hand gestures for a production AI avatar livestream. Live coordination with the CV team.
May 2025→
Alethia AI📈
SOTA Model Benchmarking Pipeline
Benchmarked generative models via advanced prompt templating and fault tolerance to recover from erroneous user prompts.
2026→
Alethia AI📝
Meme Gen & OCR Correction
Automated image pipeline integrating text generation and Google OCR to instantly detect and correct distorted meme text layers.
2024→
Alethia AI🕷️
YouTube Lip-Sync Data Scraper
Web UI that autonomously trawls YouTube links, automatically filter, framing and cropping optimal scenes for lip-sync training.
2025→
DressMe AI👗
Virtual Try-On Pipeline
Diffusion-based virtual try-on. Fine-tuned PHNet image harmonisation. Three.js image relighting with a virtual light source.
Jun 2023–Oct 2024→
DressMe AI🎨
PHNet Harmonization & Relighting
Boosted PHNet harmonization performance by augmenting custom datasets visually utilizing a robust Three.js lighting engine.
Jun 2023–Oct 2024→
DressMe AI⚙️
Multi-GPU Distributed Training
Large-scale multi-GPU distributed training pipelines for dataset preparation and model training for the try-on system.
Jun 2023–Oct 2024→
Independent📰
Agentic AI News Pipeline
LangChain-based agentic workflow for automated daily AI news updates with context. Integrated with automated posting on X and Hashnode.
Personal Project→
Independent🌐
WebGL Real-Time Lip-Sync
Browser-based real-time lip-sync using WebGL. Automated video creation pipeline built with Remotion.
Personal Project→
Independent🌍
English–German NMT Model
Neural machine translation using LSTM encoder-decoder with attention mechanism for English-to-German translation.
Academic→
Independent📈
TMDB Box Office Prediction
Kaggle competition entry using Random Forests and feature engineering for movie revenue prediction.
Kaggle→
Independent🎵
Neural Network Music Generation
Generative RNN trained on MIDI data to produce new musical sequences. Temperature-based sampling for creativity control.
Academic→
Research & Blogs

Technical Writing & Research

Technical deep dives, architectural explainers, and thoughts on AI/ML published on Medium.

IEEE Publication ↗

VarDenGrid: Variable Density Grid-Based Initialisation for Automatic Parameter Estimation

Published at IEEE INDICON 2021. A novel clustering algorithm designed to handle variable-density datasets while eliminating the need for manual parameter tuning.
Deep Learning ↗

Enhancing PHNet: How I fine-tuned the #1 Image Harmonization Model to make human skin look real

A deep dive into adapting the PHNet model to accurately blend human skin tones in generative outputs, avoiding the common plastic look.
AI Philosophy ↗

The Power of Why: Understanding the Motivation Behind AI

Exploring the core driving forces of artificial intelligence from an engineering mindset, questioning the fundamental purpose behind building these systems.
Computer Vision ↗

Unveiling the Magic: Understanding Various Face Detection Algorithms

A technical breakdown and comparison of historic and modern face detection algorithms—from Haar Cascades to deep-learning-based MTCNN and YOLO face models.
Agentic AI ↗

I Built a Zero-Cost AI Agent That Reads the Internet Every Morning and Publishes a Newsletter

It scrapes the web for the top AI stories, fact-checks each one, deep-researches the verified ones in parallel, then writes a full newsletter and X thread — and publishes everything automatically to Hashnode and X via browser automation. Five specialized LangGraph agents, each with a distinct job. Groq Llama 3.3 as the brain. No paid APIs. No manual steps.

Certification

Education

Academic Background

🏫
Class 10 (Secondary)
Kendriya Vidyalaya ARC Doom Dooma
Passed out: 2011
CGPA 9.2 / 10
🏫
Class 12 (Senior Secondary)
Kendriya Vidyalaya ARC Doom Dooma
Passed out: 2013
Percentage: 80.8%
📚
Bachelor of Technology — Computer Science & Engineering
Tezpur University, Assam
Jul 2014 — May 2018
GPA 7.36 / 10
🎓
Master of Technology — Computer Science & Engineering
IIEST Shibpur (Deemed University), West Bengal
Jul 2019 — Aug 2021
GPA 9.03 / 10
🔬
Doctor of Philosophy — Computer Science & Engineering
Indian Institute of Technology, Delhi
Jan — Mar 2022
Commenced
Certifications
Let's Talk

Let's Build
Something Real.

Open to AI/ML, Computer Vision, and Generative AI roles.

rohitpaul97@gmail.com LinkedIn GitHub Download Resume