I Built a Full Retail Analytics Platform in 1 Week Using Genie Code — Here's Everything That Happened

There's a specific kind of scepticism that data engineers have about AI-assisted development tools. We've seen too many demos where everything works perfectly on a clean five-column CSV. Production reality is messier — multiple source systems, schema mismatches, data quality failures, CDC complexities, and business logic that doesn't fit neatly into a prompt. So when I set out to build a full retail analytics POC using Databricks' newest capabilities — Genie Code, Lakeflow Connect, and Spark Declarative Pipelines — the goal was honest validation, not a polished demo.

The scope was deliberately non-trivial. Three heterogeneous sources. Six to ten tables. A full Medallion Architecture with data quality enforcement. A star schema Gold layer. A semantic layer. A dashboard. And a natural language analytics interface that a business user with zero SQL knowledge could actually use. Estimated manual development time: 20 to 30 days. Actual time with these tools: one week. Here is what happened, what worked, what surprised me, and what still needs attention.

The Stack and What Each Layer Does

The architecture flows in one direction: sources feed into ingestion, ingestion feeds the Bronze layer, Spark Declarative Pipelines transform through Silver and Gold, a Metric View sits as the semantic layer on top of Gold, and both a Databricks Dashboard and a Genie Space consume from the semantic layer. Every layer is on Databricks. No external orchestration tools, no separate ingestion platform, no BI tool — the entire stack is native.

• ▸Sources: Salesforce (CRM data), PostgreSQL (transactional database), Azure Blob Storage (flat files)
• ▸Ingestion: Lakeflow Connect for Salesforce and PostgreSQL, AutoLoader for Azure Blob
• ▸Transformation: Spark Declarative Pipelines (Bronze → Silver → Gold) built with Genie Code
• ▸Semantic layer: Databricks Metric Views with 6 measures and 15 dimensions
• ▸Consumption: Databricks Dashboard + Genie Space for natural language queries

Part 1 — Ingestion: Lakeflow Connect Changed My Expectations

I will be direct: I went into the ingestion layer expecting to spend at least three to four days. Setting up a Salesforce connector historically means dealing with API authentication, object selection, incremental extraction logic, rate limit handling, and schema change management. PostgreSQL means CDC configuration at the database level, writing or configuring a connector, and handling the replication slot. I was prepared for a slog.

Lakeflow Connect is Databricks' managed ingestion service with native connectors for enterprise applications and databases — including Salesforce Sales Cloud and PostgreSQL — built directly into the platform. The setup is point-and-click in the Databricks UI. You authenticate the source, select which tables or objects to replicate, choose your ingestion mode, and configure your schedule. Lakeflow Connect handles everything else: the initial full load, incremental updates using Change Data Capture, Slowly Changing Dimension Type 2 history tracking if you want it, schema evolution as source schemas change, and observability with built-in alerting.

For PostgreSQL, Lakeflow Connect used CDC under the hood to capture every insert, update, and delete with low latency — no need to configure replication slots manually or manage a separate connector service. For Salesforce, it connected natively to the Sales Cloud API and replicated the selected objects directly into Bronze Delta tables. The entire ingestion setup for both sources — authentication, object selection, CDC configuration, and initial load — took under 30 minutes.

For the Azure Blob Storage source, I used AutoLoader — Databricks' incremental file ingestion mechanism. AutoLoader continuously monitors a storage path, detects new files as they land, infers schema automatically, and processes them incrementally into a Delta table. It handles schema evolution, malformed record recovery, and exactly-once processing semantics. The AutoLoader notebook for this POC was generated entirely by Genie Code in Agentic mode, which I'll cover in the next section.

Part 2 — Genie Code: What Agentic Mode Actually Does in Practice

Genie Code is Databricks' AI agent, launched in March 2026. It has two modes. General Chat mode is a standard code assistant — you ask it questions, it suggests code, you copy what you need. Agentic mode is something meaningfully different. In Agentic mode, Genie Code can plan a multi-step solution, search your Unity Catalog tables, write code, execute it, read the output, identify errors, propose fixes, apply them, and re-run — autonomously, with approval checkpoints at key decision points. It adapts to wherever you are in Databricks: in the Lakeflow Pipelines Editor it focuses on pipeline development; in a notebook it focuses on data exploration and analysis.

I used Genie Code in Agentic mode inside the Lakeflow Pipelines Editor to build the entire transformation pipeline. The interaction was conversational. I described the Bronze tables, described what I wanted the Silver layer to look like, described the star schema I needed in Gold. Genie Code asked clarifying questions when my description was ambiguous — which grain should the fact table be at, should the date dimension cover a specific range, which columns need null enforcement versus soft warnings in the quality expectations. Then it went to work.

Specifically, Genie Code in Agentic mode did the following without manual coding on my part: searched the Bronze tables to understand the actual data shapes, generated each Silver dataset definition with appropriate transformations, added data quality expectations based on what it found in the data, ran the pipeline, identified a schema mismatch on one table, proposed and applied the fix, re-ran, and confirmed success. For the Gold layer it generated all five dimension tables and the central fact_sales table with correct foreign key relationships and grain definitions. The AutoLoader notebook for Blob Storage was also produced this way — I described the source path and destination, Genie Code generated the notebook, ran it, found a minor column type issue, and fixed it automatically.

The honest time comparison: building the equivalent pipeline manually — writing the Spark Declarative Pipeline definitions, adding quality expectations, debugging the schema issues, building the star schema — would realistically have taken five to eight days for someone familiar with the stack. With Genie Code in Agentic mode, the total time from first prompt to validated pipeline was under two days. That is a real and significant compression.

Part 3 — Spark Declarative Pipelines: The Transformation Engine

All three transformation layers — Bronze, Silver, and Gold — were implemented using Spark Declarative Pipelines, formerly known as Delta Live Tables. Rather than writing imperative Spark jobs that you have to orchestrate and manage manually, Declarative Pipelines let you define what each dataset should look like and the platform figures out the execution order, handles incremental processing, manages checkpoints, and provides built-in observability.

For the Silver layer specifically, the data quality expectations were the most valuable feature. You define expectations declaratively — this column must not be null, this value must be within this range, the count of records must match the source within a tolerance — and Spark Declarative Pipelines enforces them on every run. Records failing critical expectations are quarantined into a separate bad records table rather than silently corrupting the Silver layer. Genie Code added these expectations automatically based on what it observed in the Bronze data during the agentic planning step.

The Gold layer star schema — one fact_sales table and five dimension tables — was also implemented as a Declarative Pipeline. This means the Gold layer is not just a one-time transformation; it updates incrementally with every new batch of Silver data, maintains referential consistency, and has the same quality enforcement as Silver.

Part 4 — Metric Views: Solving the KPI Definition Problem

Most analytics platforms end up with a proliferation of SQL views — one per dashboard, one per report, one per team. Three months in, you have thirty views and three different definitions of 'Total Revenue' depending on which view you query. This is not a technical failure, it's an organisational one: when creating a new view is the path of least resistance, people create new views.

Databricks Metric Views are a semantic layer object that addresses this directly. You define measures — quantitative calculations like total revenue, average order value, return rate — and dimensions — the attributes you can slice by, like product category, customer segment, country, date — once in a single governed object. Every downstream consumer, whether a dashboard, a Genie Space, or an ad-hoc SQL query, references the same definitions. For this POC, 6 measures and 15 dimensions were defined covering the key retail analytics requirements.

• ▸Measures: Total Revenue, Total Orders, Average Order Value, Units Sold, Return Rate, Revenue Growth Rate
• ▸Dimensions: Date (year/quarter/month/week), Product, Product Category, Customer, Customer Segment, Geography (country/region/city), Channel, Sales Rep, and more

The practical implication is that when a business user asks Genie Space 'what was total revenue last quarter' and a data analyst queries the dashboard for the same number, they get the same answer — because both consume from the same Metric View definition. That consistency is what builds trust in a data platform.

Part 5 — Genie Space: Natural Language Analytics That Actually Works

Genie Space is the natural language analytics interface in Databricks. A business user opens it, types a question in plain English, and receives an answer — a result table, optionally with a visualisation. No SQL, no dashboard navigation, no waiting for an analyst. The accuracy and usefulness of Genie Space depends heavily on how well it is configured, which is something data engineers control.

The configuration for this POC included: connecting Genie Space to the retail Metric Views, adding sample questions with verified SQL expressions so Genie can answer common questions accurately without reasoning from scratch each time, defining join rules to control how tables can be combined and prevent incorrect cross-joins, and adding business term definitions so Genie understands domain vocabulary like 'high-value customer' or 'top performer'.

The sample questions and their verified SQL expressions are the most important configuration step. Genie uses these as reference anchors — when a user asks a question similar to a configured sample, Genie can generate the SQL confidently and accurately. Some of the questions configured for this space:

• ▸What was the total revenue for Q1 2025 compared to Q4 2024?
• ▸Which products had the highest sales volume last month?
• ▸Show me the revenue distribution across countries for this year
• ▸Which customer segment contributed the most to revenue in the last quarter?
• ▸What are the top 10 performing regions by order count this year?
• ▸How did weekly sales trend across Q2 2025?

In practice, business users could ask variations of these questions — different time periods, different product filters, different geographic breakdowns — and get accurate answers. The Genie Space also handles follow-up questions within the same conversation, so 'now show me just the top 5' or 'break that down by product category' work without starting a new query.

What This POC Actually Proved

The time compression was real. A full retail analytics platform — three sources, Medallion Architecture, star schema, semantic layer, dashboard, natural language interface — in one week versus an estimated 20 to 30 days manual. That is not a small efficiency gain. But the more interesting finding was about where the time went and where it didn't.

• ▸Time saved by Lakeflow Connect: ingestion setup for two sources that would normally take 3–4 days took under an hour. No connector code, no CDC configuration, no schema management scripts
• ▸Time saved by Genie Code: the transformation layer that would have taken 5–8 days of manual pipeline coding took under 2 days with the agentic loop handling generation, execution, and error fixing
• ▸Time that was not meaningfully compressed: business logic review, data quality validation, Genie Space configuration, and semantic layer design still required careful human attention — the tools accelerated execution but did not replace thinking
• ▸Unexpected benefit: Genie Code's error fixing loop was faster than my own debugging on two occasions — it identified a schema drift issue and a null propagation problem that I would have spent time hunting manually

What Needs More Attention Before Production

• ▸Genie Code's generated business logic should always be reviewed — it is accurate on structure and boilerplate but may not capture nuanced business rules without explicit guidance
• ▸Genie Space accuracy degrades on highly complex multi-table questions without carefully crafted sample SQL — invest time in the configuration, it pays off
• ▸Lakeflow Connect is relatively new for some connectors — the PostgreSQL and Salesforce connectors work well but monitor for updates as they mature toward GA
• ▸Metric Views are a newer Databricks feature — test all consumer queries against them thoroughly before business users rely on them in critical decisions

Final Thoughts

The Databricks platform in 2026 is genuinely different from what it was two years ago. Lakeflow Connect, Genie Code, Spark Declarative Pipelines, Metric Views, and Genie Space together form a stack where the majority of the execution work — connector setup, pipeline boilerplate, schema management, error debugging — can be handled by the platform itself. What remains for the data engineer is the higher-level work: understanding the business domain, designing the data model, reviewing what the AI generates, and ensuring quality and governance throughout.

That is a genuinely better use of a data engineer's time than writing Spark boilerplate and debugging connector schemas. This POC convinced me of that. The tools are not perfect — Genie Code needs guidance, Genie Space needs careful configuration, and everything needs review before production — but the direction is clear. The question for every data team is how quickly they adopt these capabilities before the teams that do have a significant velocity advantage.