Back to all projects
AI / LLM
AMEX · 2024 – Present

AI-Assisted Data Engineering Workflow

Embedded LLM tooling into the data platform team's daily workflow to compress DAG development and review cycles.

The problem

DAG development, boilerplate ETL code, and SQL query tuning consumed disproportionate engineering time relative to platform growth.

Impact

Engineering productivity
Measurably improved
Review turnaround
Faster cycle time

Approach

  1. 1

    Standardized use of GitHub Copilot, ChatGPT, and Claude across the data platform team for DAG scaffolding.

  2. 2

    Automated boilerplate generation for recurring ETL pipeline patterns.

  3. 3

    Introduced AI-assisted code review to streamline SQL query optimization and catch regressions earlier.

Stack

  • GitHub Copilot
  • ChatGPT
  • Claude
  • SQL
  • Airflow