Enterprise · Data Pipelines · UX Strategy & Product Design

Reproducible data workflows, without limiting the options

Data teams spend substantial time shuttling SQL data between tools just to clean it. As the lead designer, I scoped and designed a no-code pipeline that traces every transformation and turns Excel-like interactions into SQL behind the scenes, so analysis stays reproducible without forcing anyone to write SQL.

By Colin Shiner · Lead Designer

A higher-fidelity pipeline mockup: recipe nodes trace each transformation along the top, with the resulting data table in view below. — The pipeline at higher fidelity: recipe nodes trace each transformation, with the resulting data in view below.

Background

Market research and interviews with 30 data scientists, engineers, analysts, and system administrators surfaced a pattern. Users in client companies were increasingly being asked to wear multiple hats: clearly-defined roles blurring into shared responsibilities as teams stayed productive amid employee turnover, changing technologies, and more incoming raw data than ever before.

Many depended on SQL data sources without SQL expertise themselves, so they spent substantial time shuttling data between environments just to clean and prepare it for analysis, typically moving it out of a SQL environment into one that let them use Python, Excel, or similar. Working with a consulting team, we tested potential solution areas and found an opportunity to give analytics teams a better way to turn raw inputs into actionable information.

A montage of tools data teams employ throughout the week (IDEs, code editors, dashboards, spreadsheets, and command-line panes) typifying the fragmented multi-tool stack. — Some of the tools data teams juggled across a single analysis: the multi-tool tax we were aiming to remove.

Process

I began sketching potential product solutions while a colleague from product management secured approval to develop a proof of concept. Through iterative wireframes and Figma prototypes I explored different feature sets, then worked with product managers, engineers, and product leadership to scope an MVP, one that could be built quickly while clearly demonstrating value to prospective clients. Higher-fidelity Figma mockups followed, to clarify how features and flows fit together as one coherent product.

An early sketch of the pipeline workflow (hand-drawn steps for Upload Dataset, Join Datasets, and Cleanup Data) alongside a medium-fidelity mockup showing recipe cards (Uploaded Dataset, Import Static Data, Join Datasets, Cleanup Zip Codes, Publish Dataset) connected with arrows and a data preview below. — From sketch to mockup: hand-drawn steps formalised into recipe cards, each card a reusable transformation in the pipeline.

The product

The result is a data pipeline: a no-code/low-code interface that ingests and transforms data from thousands of databases, with the option to toggle back to a SQL-friendly interface whenever the user chooses. It visually traces each transformation, and lets people build their own reproducible transformations using Excel-like interactions that we translate to SQL behind the scenes, so analysis stays reproducible without forcing everyone to write SQL in a vacuum.

A birdseye view of a data pipeline with recipe nodes connected left-to-right, showing datasets being joined and cleaned before publication. — A birdseye view: datasets join, get cleaned, and publish, built from reusable recipes.

A higher-fidelity mockup showing an action-menu dropdown opened over a multi-column chartbook grid from the company design system, listing operations like Add operations, Combine outputs, and Validate columns. — An action menu over the chartbook grid: typed operations and pipeline outputs at hand.

Extending the design system

In recent years the marketing and UX teams had been investing heavily in a company-wide design system to speed up development and give the products a consistent look. As the Pipelines mockups progressed to higher fidelity I took care to emulate that visual style while also extending the system with the new types of elements and interactions the product needed: recipe cards, action menus, and an element based on a preexisting "chartbook" component among them.

Each card on the left of the canvas represents a data transformation (rename a column, multiply two columns together, filter by criteria, and so on), and clicking a card opens its configuration options. Cards reorder by drag-and-drop, so users could rearrange how their data flowed through the pipeline with minimal re-writing.

Chartbook component spec: extending the design system to cover the new pipeline UI.

A higher-fidelity component composition mockup using placeholder text: recipe cards on the left, a configuration panel in the middle, and chartbook columns on the right. — Higher-fidelity composition: how the pieces sit together as one coherent product.

By the numbers

30: Users interviewed
1 yr: Design partnership
0: Lines of SQL required

Outcome

The concept landed: early demos had user-research participants asking, “how do I sign up for advance access?”, and internal sales and market-sizing analysis (figures confidential) validated the opportunity. The work carried forward as the foundation for a state of the art AI & data platform.

See how this fed into a 0→1 data & AI platform