Engineering blog
Technical posts from our engineers and the community. Sign in to write one, or join the discussion.
Sudip GhoshinCommunityLearn how LLM fine-tuning works, when to use it instead of RAG or prompt engineering, how LoRA and QLoRA reduce training costs, and how to build a production-ready fine-tuning pipeline.
17 min read
Engineering change data capture on Delta Lake: deduplication, sequence guards, soft deletes, SCD2 in one MERGE, rebuilding history for late data, idempotent streams.
7 min read
How Delta's transaction log and concurrency work, plus time travel, deletion vectors, liquid clustering, change data feed, Unity Catalog, Auto Loader and Lakeflow.
8 min read
The maths behind gradient boosting, the split-gain formula, histogram and leaf-wise growth, missing values, categoricals, tuning, calibration and SHAP — with code.
9 min read
Partitioning and ordering, durable producers, transactions and exactly-once limits, consumer rebalancing, retries and DLQs, schema evolution and Spark streaming on Kafka.
8 min read
How LoRA and QLoRA work, GPU memory maths, building training data, a working SFT pipeline, hyperparameters, DPO preference tuning, evaluation and multi-adapter serving.
8 min read
Linear and logistic models, k-NN, SVMs, random forests, boosting, clustering, anomaly detection, forecasting and recommenders — what each assumes and how each fails.
9 min read
Our playbook from notebook to production: repo layout, Asset Bundles, CI/CD with service principals, data testing, idempotent backfills, MLflow and operations.
6 min read
Talk to the engineers behind these posts.
Have an idea for an intelligent product?