curiosity sells a better tomorrow
just a guy on the internet
SECTOR 07
X: 1842.3
Y: -992.7
Z: 120.5
A SMALLER
INTERNET
SOMEWHERE
EST. 1998   ,    STILL EXPLORING
MAYANK SRIVASTAVA   ,    V 1.0
SCROLL DOWN
⌄
mayank.system
“I made a website instead of going to bed.”

There are no achievements hidden here. There is, however, a completely unnecessary WebGL game running behind this page.

01 / THINGS I LIKE

Pick a doorway.

The site is a collection of worlds rather than a list of credentials.

01 / ABOUT

The person
behind the systems.

A little less “professional bio”, a little more “what makes Mayank tick”: music, machines, sport, games, food, travel, and a frankly unreasonable number of browser tabs.

Mayank in the mountains
FIELD NOTE / 01 A good view, a new place, and absolutely no reason to rush back.
A QUICK READ

Hi, I’m Mayank. Curious about almost everything, competitive about a few things, and permanently one tab away from a new rabbit hole.

I like computers for the same reason I like a good strategy game: there is always another layer underneath. I enjoy coding, tinkering, figuring out how things work, trying things just to see what happens, and occasionally spending far too long making something unnecessarily polished.

Music is a big part of the background noise. So are Formula 1 weekends, Real Madrid matches, tennis, strategy games, travelling somewhere new, and stopping for food because a place looked interesting. I’m especially drawn to people and ideas with a certain mindset , the kind of persistence you see in Novak Djokovic and Cristiano Ronaldo: keep improving, stay stubborn when it matters, and somehow find another gear when everyone else thinks the job is done.

And yes, despite having an entire career around modern technology, I still open The Verge and XDA Developers basically every day like it is 2014 and I’m waiting to discover the next piece of tech that will completely change my life. It usually does not. I still check tomorrow.

Travel keeps the rest of it moving. Give me a new city, a mountain road, an unfamiliar menu or a completely unplanned afternoon and I’m interested. I like collecting experiences more than souvenirs , good songs, ridiculous sports moments, strange little discoveries, great meals and stories that start with “we probably shouldn’t have gone there” are generally enough.

IN SHORTBuild things. Watch races. Back Madrid. Learn something. Find good food. Repeat.
♪ MUSIC🏎 F1⚽ REAL MADRID♟ STRATEGY GAMES⌘ COMPUTERS⌁ CODING✈ TRAVEL🍜 FOOD
THINGS I’M INTO
01
Computers & coding

I can lose an afternoon to a machine, a weird bug, a new tool, or an idea that started with “I wonder if…”

02
Music

Blues, old records, modern discoveries, playlists, radio rabbit holes and songs that somehow become part of a particular memory.

03
Formula 1

Cars are great. Strategy is better. Tyres, timing, engineering, tiny margins and the occasional decision that makes everyone yell at the screen.

04
Real Madrid

Big nights, ridiculous standards, beautiful football and the belief that a match is never really over until it is over.

05
Tennis

Especially the Djokovic school of mentality: stay in the fight, solve the problem in front of you, and make the next point matter more than the last one.

06
Strategy games

I like systems where patience, planning and one slightly questionable decision can completely change the outcome.

07
Travel & food

New places are best explored slowly, preferably with a good walk, a questionable detour and something local on the plate.

08
Tech rabbit holes

The Verge, XDA, old forums, new hardware, strange software and “just five minutes” of reading that somehow becomes an hour.

THE SHORT VERSION
Give me a computerI’ll probably find something to build, automate, break, improve or unnecessarily redesign.
Give me a raceI’ll start talking about strategy. Give me a second race and I’ll have opinions about tyre degradation.
Give me a matchMadrid for the drama. Tennis for the mentality. Both for the completely rational decision to care way too much.
Give me a playlistThere is a good chance I’ll listen to the same song repeatedly until it becomes permanently attached to a memory.
Give me a mapI’d rather explore a new place, take the scenic route and find a good local meal than perfectly follow an itinerary.
Give me the internetI will somehow end up on The Verge, XDA Developers, an ancient forum thread or a 2012 review of a device nobody remembers.
Give me an ideaIf it sounds interesting enough, there is a decent chance I’ll turn it into a project , even if nobody asked me to.
02 / MUSIC

The soundtrack
department.

A small corner of the internet for the music that follows me around , old blues, modern discoveries, late-night listening, road-trip songs and anything that earns another play.

Black Muddy River ,  Dead & Company live album art
CURRENTLY PLAYING

Black Muddy River

Jiffy Lube Live, Bristow, VA 8/18/21 (Live) · Dead & Company

The kind of song that makes you stop doing whatever you were doing for a minute.

THE SHORT VERSION

Good music doesn't need a category.

My playlists jump around quite a bit. Blues, guitar-driven stuff, feel-good records, newer releases, songs for travelling and the occasional track that gets played far more times than it probably deserves.

These are the playlists I actually keep around. No elaborate genre taxonomy , just music I like, music I discover, and music that fits a particular moment.

THE ROTATION

Playlists I keep coming back to.

Open any playlist and let Apple Music take it from here.

01

B.B. King Essentials

Open in Apple Music ↗

Blues, soul and the kind of guitar playing that makes you pay attention.

02

Chill & Feel Good Songs

Open in Apple Music ↗

Easy listening, good moods and songs that make an ordinary day feel a little better.

03

J.J. Cale Essentials

Open in Apple Music ↗

Laid-back grooves, warm guitars and the sort of songs that reward slowing down.

04

ACM New Music Friday

Open in Apple Music ↗

New releases and fresh finds , because the old favorites still need some competition.

05

John Mayer Essentials

Open in Apple Music ↗

Guitar, songwriting and the particular kind of calm that works equally well on a drive or at midnight.

TAWANG 2026 · 01 / 14
Tawang 2026 ,  Mountain Memories page 1 of 14
TAWANG 2026 · MOUNTAIN MEMORIES
full-screen photobook · use ← →
03 / WORK + PROJECTS

A career built
by going deeper.

From analytics to data engineering to GenAI, the path has been less about changing direction and more about moving closer to the systems underneath the question.

CAREER TRANSITIONJUN 2022 → PRESENT

Started with the question.
Moved toward the system.

The progression has been deliberate: first learning how healthcare claims data behaves, then engineering the pipelines and platforms behind it, and now building cloud and GenAI systems that sit on top of that foundation.

That means the work is not just a list of tools. It is a shift in scope: from answering individual business questions, to making those answers repeatable, to designing the infrastructure that lets other people ask better questions faster.

MAR 2026 · PRESENT

Senior Data Engineer

Cotiviti

Drove Hadoop-to-OCI and Databricks modernization across data, analytics and AI use cases. Built production GenAI applications using RAG, GraphRAG, Oracle 26ai, Property Graph, AI Vector Search, OCI Generative AI, OpenShift and Kubernetes. Reduced time to answer from three hours to under one minute. Built a 141-file knowledge pipeline across four claims streams, reducing engineer ramp time from six weeks to under two weeks, and mentored four engineers.

SEP 2024 · MAR 2026

Data Engineer

Cotiviti

Architected rule-governance intelligence with PySpark, Hive, Impala and Power BI, evaluating 38 rule-behavior dimensions and surfacing $10M+ annual savings opportunities. Modernized 25+ legacy Hive, Impala and Pig workloads to PySpark, Spark SQL and Parquet, cutting runtime by up to 90%. Built an ML-driven data-health platform covering 374 client environments across five years and $4B+ in paid healthcare claims.

JUN 2022 · AUG 2024

Data Analyst

Cotiviti

Analyzed healthcare claims data with SQL, Hive, Impala, Python and HDFS, delivering 100+ ad hoc analyses across 300+ tables. Engineered a 4,400-line Impala SQL analytics engine across 28 tables with five TaxGroup TIN analysis variants. Automated recurring reporting, ETL, QC and reconciliation with Python, SQL, Oozie, Airflow, Kafka, Oracle, Hive and HDFS, while supporting 5+ flagship data products.

ANALYTICS→DATA ENGINEERING→PLATFORM + CLOUD→GENAI
PROJECTS FOLDER

Selected systems, opened on demand.

Seven projects, grouped by the kind of work they do. ★ marks Argos as the featured project; open any row to explore its approach, stack, scale and results.

AI Artificial intelligence DE Data engineering DA Data analytics ML Machine learning
01★ArgosAIDEMany-eyed guardian of reliable answers · RAG → GraphRAG · Oracle 26ai · Databricks · Kubernetes3 hr → <1 min+
THE SYSTEM

Ask the data without making the user understand the data estate first.

Named for Argos, the many-eyed guardian of Greek mythology, this healthcare-claims intelligence platform turns pipeline and business questions into traceable answers. It evolved from intent-routed RAG on Databricks Apps into GraphRAG on Oracle 26ai and OCI Generative AI, combining source navigation, schema-aware SQL validation and a dedicated reliability harness.

ARCHITECTURE

First generation: Databricks Apps, Unity Catalog Volumes, Foundation Model APIs, eight native LLM tools and two-phase SQL validation. Successor: Oracle Property Graph and AI Vector Search via OCI Generative AI. LangGraph StateGraph handles intent and tool routing; Langfuse provides observability; OpenShift/Kubernetes supports in-cluster deployment.

DATA + ENGINEERING

PySpark schema extraction mapped 74 tables, about 1,900 columns and 157 date formats. That metadata feeds SQL validation and constrains what the system can query.

OUTCOME

Pipeline-question lookup fell from about three hours to under one minute. The result is an enterprise application spanning retrieval, graph reasoning, data contracts, validation, observability and deployment.

RAGGraphRAGLangGraphOracle 26aiOCI Generative AIPySparkKubernetesLangfuse
Argos / RELIABILITY ENGINEERING

A harness for answers the system was never taught to anticipate.

The harness handles open-ended questions across documentation, live relational data and production scripts. It checks each draft against evidence gathered in that turn, then runs a bounded second review that can investigate and correct gaps—without adding a hardcoded rule for every past failure.

Evidence-led · bounded · observable
3Ground-truth surfacesDocumentation · live database · raw scripts
8Evidence-structural checksFindings guide the audit; they do not decide
40 / 12Draft / audit round capsAudit also has a 300-second ceiling
535Offline testsMechanism coverage, not an answer-quality score
01 / HOW ONE ANSWER IS BUILT

Read the flow from left to right. Click a step to see what happens. Argos looks up evidence, drafts an answer, checks it, runs a second review and then keeps or corrects the draft.

SOURCE 01

Curated documentation

Explains intent and definitions. Useful context, but can be incomplete or stale.

SOURCE 02

Live relational data

Authoritative for stored values and live schema details; matching resolves the user's phrasing to values that actually exist.

SOURCE 03

Production scripts

Ground truth for build logic and physical storage. Parsed for lineage; never executed by the lineage analyser.

One question, six steps

QUESTION + INTENT

The system classifies the user's request through a forced typed call, including intent and trend / coverage flags. It routes charts, trivial replies and all other questions to the appropriate path.

The core idea: judge an answer by what was actually found, not by how confident it sounds. Results are size-limited and source-tagged; previews are labelled, dead code is marked, secrets are redacted, and a search miss triggers another route rather than a claim that something does not exist.
Explore the ten engineering contributions
C1
Evidence-structural guarding

Compares answer claims with results, calls and source inventory.

C2
Action memory

Remembers prior actions, not potentially hostile results.

C3
Parse-never-run handling

Parses simple call code; never executes it.

C4
Central reasoning-cost accounting

Adds per-effort reasoning headroom in one shared client.

C5
Storage-aware lineage

Follows definitions to storage paths, including across scripts.

C6
Aim-don't-decide verification

Findings guide the audit; code does not decide or rewrite.

C7
Typed-decision control plane

Intent, verdict and memory use typed calls with recovery.

C8
Derived value resolution

Matches user wording to live values across indexed columns.

C9
Signals and knowledge growth

Trace signals propose new knowledge; people approve changes.

C10
Failure-type engineering

Tests renamed cases, variants, hostile inputs and negative cases.

Limits: one model family and corpus; small, non-deterministic samples judged by the same engineers who built the system; file-name-only hand-back detection; and no draft wall-clock budget yet. The next step is independent labelling and repeated ablation tests, not a claim of proven general accuracy.

02CLAIMS KNOWLEDGE BASEAIDE141 files · 960 chunks · FAISS embeddings · Oracle Property Graph6 wk → <2 wk+
THE SYSTEM

Turn domain knowledge into something both people and machines can navigate.

Built an enterprise knowledge layer across four claims streams. Python-generated cross-links organised 141 source files into roughly 960 retrieval chunks; embeddings and FAISS made the content semantically searchable, while an Oracle Property Graph and a Three.js view exposed relationships for exploration. The result became part of the chatbot retrieval backbone and cut new-engineer ramp time from six weeks to under two.

PIPELINE

Python-generated cross-links connected the source material and organised 141 files into roughly 960 chunks. Text embeddings represented the chunks for semantic retrieval, and FAISS provided the vector-search index. Cross-links retained relationships between source notes, so retrieval did not have to treat every chunk as an isolated passage.

GRAPH LAYER

The knowledge was also represented as an Oracle Property Graph with 244 nodes and 610 edges. A separate interactive Three.js exploration view exposed 1,146 edges. These counts describe the underlying property graph and the interactive exploration representation, respectively—not a single interchangeable graph count.

OUTCOME

The linked corpus and graph became part of the chatbot retrieval backbone. For onboarding, the searchable, connected knowledge reduced new-engineer ramp time from six weeks to under two weeks; the project made domain relationships explorable instead of leaving engineers to reconstruct them from scattered files.

PROJECT WALKTHROUGH

From source files to vector retrieval and graph navigation

Follow the data path: cross-link source files, chunk and embed content for FAISS retrieval, then expose explicit relationships through Oracle Property Graph and an interactive Three.js view.

4 STEPS · EXPLORE
141Source filesAcross four claims streams
~960Knowledge chunksOrganized with Python-generated cross-links
244 / 610Graph nodes / edgesOracle Property Graph representation
1,146Exploration edgesIn the interactive Three.js view
HOW THE WORK FITS TOGETHERSelect a step for the explanation
STEP 01
Collect the source material

141 source files across four claims streams formed the starting corpus. The project made this existing domain knowledge more navigable instead of requiring engineers to piece it together manually from isolated files.

PythonEmbeddingsFAISS vector searchOracle Property GraphThree.jsKnowledge retrieval
03MULTI-AGENT ANALYTICS PLATFORMAIDE14 services · 7 MCP servers · LangGraph · LiteLLM · Langfuse9 / 10 prod cases+
THE SYSTEM

A distributed AI platform where debugging is part of the architecture.

Core member of a five-engineer team building a 14-service multi-agent analytics platform for Sales Copilot and RFP Copilot workflows. LangGraph handled orchestration, seven MCP servers exposed tool capabilities, a two-tier LiteLLM gateway routed model requests, and Langfuse tracing helped investigate failures across service boundaries.

THE HARD PART

Two production failure modes required tracing behavior across the distributed path: a unit-scaling defect that magnified a value by 100×, and a silent failure that returned blank AI narratives. The investigation followed the data and control path through orchestration and MCP boundaries instead of assuming the component nearest the symptom was the root cause.

PRODUCTION VALIDATION

Fixes were validated against live production data rather than only synthetic tests. Nine of ten cases in the reported real-world validation set passed. During the investigation, an independent aggregation bug was also found beyond the original failure path.

ENGINEERING SIGNAL

The work crossed several operational layers: LangGraph orchestration, MCP tool-server boundaries, two-tier model routing through LiteLLM, and Langfuse tracing. It demonstrates distributed root-cause analysis and production validation—not a claim that the platform has universal 90% accuracy.

PROJECT WALKTHROUGH

Trace failures across orchestration, tool servers and model routing

A multi-agent platform spreads behavior across services. This walkthrough separates the orchestration layer, tool interface, model gateway and production validation so the debugging story is understandable without flattening it into “the model made a mistake.”

4 STEPS · EXPLORE
5Engineer teamCore team member
14ServicesDistributed multi-agent platform
7MCP serversTool-serving layer
9 / 10Production cases passedReported real-world validation sample
HOW THE WORK FITS TOGETHERSelect a step for the explanation
STEP 01
Orchestrate the workflow

LangGraph served as the orchestrator across a 14-service platform intended to support Sales Copilot and RFP Copilot workflows. The user-visible narrative depended on behavior across multiple components.

LangGraphMCP tool serversLiteLLM routingLangfuse tracingMulti-agent systemsProduction debugging
04DATA HEALTH SCORECARDDEDAMLOracle ↔ HDFS · Iceberg · 15 features/client-month · AE · STL + IF374 clients · 5 years+
THE SYSTEM

Make data quality measurable before it becomes an incident.

A weekly Oracle-vs-HDFS reconciliation and anomaly-monitoring workflow spanning 374 client environments, five years of history and more than $4B in paid-claims client-months. Apache Iceberg snapshots supported historical comparison; PySpark produced 15 features per client-month; three detector families combined Z-score deviation, a 9–6–3–6–9 Autoencoder (AE) using reconstruction error, and STL decomposition paired with Isolation Forest (IF) for seasonality-aware anomaly detection.

FEATURES

Oracle JDBC-versus-HDFS reconciliation supplied the data-health signal, with Apache Iceberg snapshots supporting historical comparisons. PySpark generated 15 features per client per month so recurring discrepancies could be represented consistently across time and across client environments. The scorecard covered 374 client environments over five years and more than $4B in paid-claims client-months.

MODELS

Three complementary detector families were combined: (1) Z-score deviation, a statistical signal for values far from their reference distribution; (2) a 9–6–3–6–9 Autoencoder (AE), using reconstruction error to flag patterns the model reconstructs poorly; and (3) STL (Seasonal-Trend decomposition using Loess) paired with Isolation Forest (IF)—the STL + IF approach—for seasonality-aware anomaly detection. STL separates trend/seasonal structure from remaining variation; Isolation Forest is an outlier-detection method. The project description does not claim a head-to-head winner.

DELIVERY

Oozie orchestrated a weekly workflow that packaged results into a self-contained HTML scorecard. The reporting process was adopted team-wide, making the output repeatable and reviewable rather than a one-off notebook or manual QC result. The documented material does not provide comparative precision/recall metrics, so the methods are described by their roles rather than an invented accuracy ranking.

PROJECT WALKTHROUGH

Reconcile source data, engineer monthly features, then run three anomaly signals

The scorecard does not rely on one detector. Oracle-vs-HDFS reconciliation establishes the signal; Iceberg snapshots preserve the historical view; PySpark derives client-month features; Z-score, AE (Autoencoder) and STL + IF (STL decomposition + Isolation Forest) contribute complementary ways to surface unusual behavior.

4 STEPS · EXPLORE
374Client environmentsCovered by the scorecard
5 yearsHistorical windowLongitudinal client coverage
$4B+Paid claims coverageAcross client-months
15Features per client-monthEngineered with PySpark
HOW THE WORK FITS TOGETHERSelect a step for the explanation
STEP 01
Reconcile the data sources

Oracle-versus-HDFS reconciliation provides the base signal. Apache Iceberg snapshots support the historical view used to compare client behavior across time.

PySparkOracle JDBCApache IcebergZ-scoreAE · 9-6-3-6-9STL + IFOozieWeekly HTML scorecard
05RULE CUSTOMIZATION INTELLIGENCEDEDA38 behavior dimensions · fork-chain lineage · 44-column score · Power BI$10M+ / year+
THE SYSTEM

Business behavior has lineage too.

Reverse-engineered the relationship between standard and client-customized healthcare claims editing rules. Instead of comparing rule names alone, the workflow traced client fork chains, measured code-scope drift across 38 behavioral dimensions, and converted the findings into a 44-column scoring table and Power BI lineage view for ranked review.

ANALYSIS

The pipeline traced fork chains from standard rules to client-specific versions and measured code-scope drift across 38 rule-behavior dimensions. A Python clustering and similarity prototype was used to explore behavioral groupings before the production workflow was hardened. The source does not enumerate the 38 dimensions, so the portfolio does not invent a feature-by-feature list.

PRODUCT

The resulting 44-column scoring table structured the rule-level evidence, while a Power BI lineage view connected standard/custom relationships to review. False positives were hardened before the output was treated as a ranked decision aid, rather than leaving analysts with an unfiltered similarity list.

OUTCOME

The workflow surfaced more than $10M in annual recoverable savings opportunities and reduced a roughly three-day manual review cycle to a same-day ranked report. The $10M+ is an identified opportunity, not booked or guaranteed savings.

PROJECT WALKTHROUGH

Trace rule forks, measure code drift, and turn comparison into a review queue

Rule names do not fully describe behavior. The workflow follows standard-to-client fork lineage, evaluates 38 behavior dimensions, uses a Python similarity/clustering prototype to explore groupings, then packages the hardened result for review.

4 STEPS · EXPLORE
38Behavior dimensionsUsed to evaluate rule differences
44 columnsScoring tableStructured rule-level output
$10M+Annual opportunity surfacedRecoverable savings opportunities
~3 days → same dayReview turnaroundManual cycle to ranked report
HOW THE WORK FITS TOGETHERSelect a step for the explanation
STEP 01
Compare standard and custom rules

The analysis reverse-engineered relationships between standard rules and client-specific forks, tracing fork chains and measuring code-scope drift instead of relying on names alone.

PySparkHiveRule-fork lineageCode-scope driftPython similarityClustering44-column scorePower BI
06TIN EXPOSURE ANALYTICS ENGINEDADE4,400-line Impala SQL · 28 tables · 12 sections · Prod-to-RVA / Prod-to-ProdTop 20% focus+
THE SYSTEM

Find the exposure. Then decide where to look.

Engineered a roughly 4,400-line Impala SQL analytics engine across 28 tables and 12 sections to measure provider Tax Identification Number (TIN) claims exposure touched by editing rules. It produces two distinct benchmark views, supports business-specific slices, and ranks the highest-priority opportunities instead of treating every provider TIN as equivalent.

BENCHMARKS

Prod-to-RVA compares production financials against Rule Value Addition (RVA) projections. Prod-to-Prod benchmarks a client’s TIN against other clients running the same rule. The first lens compares realised production financials with projected rule value; the second adds a peer comparison under the same rule. They answer different questions and should not be treated as interchangeable measures.

SLICES

The analysis can be segmented by Line of Business (LOB), State, Product and Claim Type, with specialty-specific views for more focused investigation. These slices let analysts examine where the exposure concentrates rather than relying only on a single rolled-up total.

DECISION LAYER

A Top-20% prioritization view ranks opportunities by annualized savings so analysts can focus on the highest-priority slice. The output is a ranking for investigation; exposure touched by an editing rule is not itself proof that every associated amount is recoverable.

PROJECT WALKTHROUGH

Measure rule-linked TIN exposure, benchmark it two ways, and prioritize follow-up

The Impala SQL engine turns provider-level TIN exposure into a comparable analysis across 28 tables and 12 query sections. Prod-to-RVA and Prod-to-Prod provide different benchmarks; business slices then support a Top-20% queue based on annualized-savings opportunities.

4 STEPS · EXPLORE
~4,400Lines of Impala SQLAnalytics engine
28TablesAcross the analysis
12SectionsOrganized query logic
Top 20%Priority viewHighest-priority opportunities
HOW THE WORK FITS TOGETHERSelect a step for the explanation
STEP 01
Map exposure touched by rules

The engine measures provider TIN claims exposure associated with editing rules. The output is an analysis of exposure connected to rule behavior, not a claim that every amount is recoverable.

Impala SQLProvider TIN exposureProd-to-RVAProd-to-ProdLOB / State / Product / Claim TypeAnnualized-savings ranking
07LEGACY SQL + PIG → PYSPARK MODERNIZATIONDE25+ Hive / Impala / Pig workloads · PySpark · Oozie · Sqoop · React analyzerUp to 90% faster+
THE SYSTEM

Replace the ceiling. Keep the business moving.

Modernized more than 25 legacy Hive, Impala SQL and Pig workloads to PySpark using Spark SQL and Parquet. The work also re-platformed Oozie workflows, automated Oracle EDW-to-HDFS extraction with Sqoop, and added a custom React Spark performance analyzer to make runtime behavior easier to inspect and tune.

MODERNIZATION

More than 25 legacy Hive, Impala SQL and Pig workloads were translated to PySpark and Spark SQL, with Parquet used in the modernized data path. Oozie workflows were re-platformed around the converted workloads, so the work changed the scheduled operating flow as well as the query language.

TOOLING

Sqoop automation handled recurring Oracle enterprise data warehouse (EDW)-to-HDFS extraction, reducing the need for repeated manual data pulls. A custom React-based Spark performance analyzer made runtime behavior easier to inspect and tune; it served as an observability aid for the migration rather than a replacement for Spark execution itself.

OUTCOME

Reported runtime improvements reached up to 90%, and automation eliminated about eight hours per week of manual data pulls. “Up to 90%” is the maximum reported improvement, not a promise that every migrated workload improved by that amount.

PROJECT WALKTHROUGH

Modernize query workloads, re-platform schedules, automate extraction and inspect runtime

This was more than a syntax conversion. The project combined Hive/Impala/Pig-to-PySpark migration with Spark SQL and Parquet, Oozie workflow changes, Sqoop-based Oracle EDW-to-HDFS extraction, and a custom React analyzer for runtime inspection.

4 STEPS · EXPLORE
25+WorkloadsModernized
Up to 90%Runtime reductionReported maximum improvement
~8 hours / weekManual effort avoidedRecurring data pulls
Oracle EDW → HDFSAutomated extractionSqoop-based workflow
HOW THE WORK FITS TOGETHERSelect a step for the explanation
STEP 01
Translate legacy processing

More than 25 Hive, Impala SQL and Pig workloads were modernized to PySpark, using Spark SQL and Parquet in the updated data path.

PySparkSpark SQLParquetHiveImpalaPigOozieSqoopReact Spark analyzer
04 / JOURNAL

Things worth
writing down.

Technical notes, experiments, internet archaeology and the occasional extremely unnecessary deep dive.

NOTE 01

How I ended up building a GraphRAG system

The path from “I should probably learn this” to something that actually shipped.

↗
NOTE 02

Production bugs are excellent teachers

Why live data keeps winning arguments that code review thought it had already won.

↗
NOTE 03

Making old pipelines less embarrassing

Migration, profiling, measuring, then resisting the urge to rewrite everything.

↗
NOTE 04

The strange art of finding old radio shows

A little digital archaeology project for anyone nostalgic about internet-era radio.

↗
EDITOR'S NOTE

Not everything needs to be useful.

Some things are just interesting. That is a pretty good reason to learn them, build them, photograph them, or spend an embarrassing number of tabs researching them.

“The internet is most fun when it feels a little bit like a room someone actually lives in.”
05 / MISC

The drawer
of side quests.

The things that never quite earn a category, but somehow keep becoming a category.

◈

Internet archaeology

Old websites, old radio archives, forgotten interfaces and strange little corners of the web.

↗
✦

Interface obsessions

Spacing, motion, microinteractions, type, and the suspicious belief that one more polish pass will fix everything.

↗
∿

Midnight experiments

Small scripts, visualizations, prototypes and “wait, can I make this?” projects.

↗
⊙

Things I recommend

Films, tools, books, sites, songs, places and anything that deserves an aggressively enthusiastic link.

↗
06 / SAY HI

Glad you
stopped by.

For work, collaboration, interesting ideas, a good song, a strange internet find, or just a “hey, this is cool” message.