{"id":116030,"date":"2026-02-24T11:37:10","date_gmt":"2026-02-24T11:37:10","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2026\/02\/24\/how-omnisend-embedded-ai-into-the-data-lifecycle\/"},"modified":"2026-02-24T11:38:11","modified_gmt":"2026-02-24T11:38:11","slug":"how-omnisend-embedded-ai-into-the-data-lifecycle","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2026\/02\/24\/how-omnisend-embedded-ai-into-the-data-lifecycle\/","title":{"rendered":"How Omnisend embedded AI into the data lifecycle"},"content":{"rendered":"

<\/a>\n
<\/p>\n

\n
\n
\n \n \n <\/svg><\/p>\n

Learn summarized model with<\/p>\n<\/p><\/div>\n<\/p><\/div>\n

In 2025, there may be maybe no group beneath the solar that isn\u2019t making data-driven choices, or at the very least claiming to. At Omnisend, knowledge isn\u2019t only a declare \u2014 it\u2019s the muse upon which the remainder of the home is constructed.<\/p>\n

DataOps & Insights<\/strong> is the supply of reality for our choices: operational reporting, predictive analytics, and self-serve knowledge. Our CI\/CD pipeline enforces high quality so product groups can transfer quick with out breaking belief.<\/p>\n

Nonetheless, after not too long ago becoming a member of the crew as a Software program Engineer, I observed a friction widespread to many knowledge groups: the mechanics of the work had been slowing down the logic. We would have liked to optimize not simply our code, however our workflows. After intensive trials, we recognized high-ROI functions for Giant Language Fashions (LLMs) that compress our time-to-insight from days to minutes.<\/p>\n

Right here\u2019s how we did it.<\/p>\n

1. The analytics hole: Velocity vs. high quality<\/h2>\n

The actual problem in fashionable knowledge engineering isn\u2019t writing SQL \u2014 it\u2019s closing the hole between a pointy enterprise query and a reliable reply earlier than the chance window closes.<\/p>\n

At Omnisend, we realized that whereas our logic was sound, the \u201cchores\u201d of knowledge modeling had been making a bottleneck.<\/p>\n

The friction: Context switching and boilerplate<\/h3>\n

Constructing a strong knowledge mannequin requires fixed context switching: leaping between dbt conventions, YAML configurations, testing suites, and documentation requirements. We confronted repetitive scaffolding duties throughout each layer of our transformation pipeline (staging \u2192 dims \u2192 details \u2192 marts).<\/p>\n

Each context change launched an opportunity for error, and sustaining consistency throughout environments turned more and more fragile. We would have liked a approach to automate the rigorous, repetitive elements of the job so our analysts may concentrate on structure somewhat than typing.<\/p>\n

The answer: Context-aware modeling with Cursor<\/h3>\n

We turned to Cursor<\/strong>, an AI-powered code editor. In contrast to commonplace autocomplete instruments, Cursor indexes our total repository, permitting it to know the particular context of our undertaking construction, knowledge lineage, and naming conventions.<\/p>\n

We arrange the surroundings to help the AI:<\/p>\n

    \n
  • Repo Indexing:<\/strong> Cursor listed our knowledge fashions and documentation, giving it a \u201cmap\u201d of our knowledge warehouse<\/li>\n
  • Guardrails and prompts:<\/strong> We established well-scoped prompts aligned with our SQL model<\/li>\n
  • Inline critiques:<\/strong> The AI flags anti-patterns \u2014 like CTEs that break incremental fashions or fan-out joins \u2014 earlier than<\/em> a Pull Request (PR) is even raised<\/li>\n<\/ul>\n

    Implementation: From hours to minutes<\/h3>\n

    With these guardrails in place, the workflow shifted dramatically. When an analyst defines a enterprise requirement, Cursor generates the preliminary knowledge fashions in seconds. It selects acceptable supply tables, generates recordsdata in right undertaking paths (staging\/dims\/details), and even pulls column descriptions to auto-populate documentation.<\/p>\n

    What used to take hours of handbook file creation is now accomplished in minutes. The analyst\u2019s position shifts from \u201cauthor\u201d to \u201creviewer.\u201d<\/p>\n

    A observe on hallucinations:<\/strong> It\u2019s necessary to be real looking: the instrument isn\u2019t good. It makes errors when the token chance sequence will get \u201cconfused.\u201d Nonetheless, getting 90% of the work accomplished immediately permits us to spend our vitality on the ultimate, essential 10% of validation.<\/p>\n

    The impression<\/h3>\n
      \n
    • Growth velocity:<\/strong> A 2-5x improve<\/strong> in mannequin supply pace by templating YAML and take a look at creation<\/li>\n
    • Improved consistency:<\/strong> SQL and YAMLs now observe a strict commonplace, decreasing knowledge incidents.<\/li>\n
    • Higher traversability:<\/strong> The AI enforces a constant hierarchy (staging > dims > details > marts<\/em>), making the codebase simpler to navigate, perceive, and use when creating a brand new knowledge mannequin<\/li>\n<\/ul>\n

      2. Fewer evaluate cycles, fewer incidents<\/h2>\n

      As soon as modeling is completed regionally in Cursor and a PR is raised, the workflow shifts from creation to validation. That is the place we hand the baton to Gemini Code Help.<\/p>\n

      The problem: The peer evaluate bottleneck<\/h3>\n

      Peer critiques are essential for high quality, however they will develop into a bottleneck. A human reviewer \u2014 particularly one from a distinct product group \u2014 may miss delicate deviations from our dbt model information or miss out on non-optimal BigQuery features.<\/p>\n

      We confronted widespread ache factors:<\/p>\n

        \n
      • Context blindness:<\/strong> Struggling to know cross-file context in massive diffs<\/li>\n
      • Fashion drift:<\/strong> Inconsistent formatting making diffs more durable to learn<\/li>\n
      • Logic gaps:<\/strong> Lacking delicate enterprise logic breaks (e.g., attribution order adjustments) that look syntactically right however are functionally mistaken<\/li>\n<\/ul>\n

        The answer: Gemini Code Help (with strict tuning)<\/h3>\n

        We deployed Gemini Code Help as our first line of protection. It summarizes diffs by intent, checks in opposition to a repo-specific model information, and proposes concrete fixes.<\/p>\n

        Nonetheless, out of the field, the AI was noisy. To make it helpful, we needed to arrange the reviewer similar to we arrange the author:<\/p>\n

          \n
        1. Noise discount:<\/strong> We tightened the .gemini\/config.yaml to prioritize essential findings over nitpicks<\/li>\n
        2. Context injection:<\/strong> We added a .\/gemini\/styleguide.md file containing our particular dbt conventions and governance checks<\/li>\n<\/ol>\n

          Actual-rorld optimization: The story of three CTEs<\/h3>\n

          The worth of a second AI opinion turned clear throughout a current refactor. We had a mannequin with three duplicated Frequent Desk Expressions (CTEs).<\/p>\n

          Cursor (the author) flagged them however instructed an \u201cif it ain\u2019t damaged, don\u2019t repair it\u201d strategy, warning that unioning may be slower.<\/p>\n

          While Gemini (the reviewer)<\/strong> flagged the identical duplication, however advisable a concrete optimization: consolidating them into one union with a single unnest\/be a part of.<\/p>\n

          We examined the Gemini-suggested refactor. The end result was a ~50% discount in runtime.<\/strong> This interaction is essential: the drafting AI prioritized pace, whereas the reviewing AI prioritized structure.<\/p>\n

          The impression<\/h3>\n
            \n
          • 30\u201340% fewer evaluate cycles:<\/strong> Gemini catches syntax and magnificence points earlier than a human sees them<\/li>\n
          • 15\u201325% discount in logical errors:<\/strong> Fewer post-merge defects tied to inconsistent logic<\/li>\n
          • Automated governance:<\/strong> The assistant flags PII points and validates source-of-truth tables robotically<\/li>\n<\/ul>\n

            3. Fixing knowledge discovery: The \u201cthe place is X?\u201d downside<\/h2>\n

            As our Superset surroundings scaled to 1000’s of belongings, it turned a sufferer of its personal success. A easy query like \u201cThe place can I discover our month-to-month recurring income chart for M-segment shoppers?<\/em>\u201d required deep platform information or a ping to the info crew.<\/p>\n

            The answer: Embed, index, retrieve<\/h3>\n

            We embedded a Chainlit<\/strong> chatbot straight into the Superset UI.<\/p>\n

              \n
            1. Ingestion:<\/strong> A each day automated pipeline (through Dagster) extracts metadata from each dashboard and chart<\/li>\n
            2. Indexing:<\/strong> Metadata is synced to a vector information base on OpenAI<\/li>\n
            3. Retrieval:<\/strong> Chainlit responds by way of the OpenAI Assistant API, returning ranked belongings with direct hyperlinks when obtainable, or suggesting the place outcomes could also be discovered.<\/li>\n<\/ol>\n

              All of it comes right down to context<\/h3>\n

              The facility of this strategy is knowing knowledge relationships. A marketer not too long ago requested: \u201cHow lengthy, on common, does it take retailers to activate kinds from the time of creation?<\/em>\u201c<\/p>\n

              No pre-built dashboard answered this. Nonetheless, the Assistant analyzed the intent and appropriately recognized the related dataset and columns<\/strong> wanted to calculate the reply. It remodeled a \u201cno outcomes\u201d useless finish right into a self-serve win.<\/p>\n

              The impression<\/h3>\n
                \n
              • Silence is golden:<\/strong> A 25\u201340% drop<\/strong> in \u201cThe place is X?\u201d pings to the DataOps crew<\/li>\n
              • Pressured hygiene:<\/strong> As a result of the bot depends on metadata, \u201cundocumented\u201d turned \u201cinvisible,\u201d incentivizing the crew to undertake higher documentation requirements<\/li>\n<\/ul>\n

                4. Scaling EDA: 76 hours of video in minutes<\/h2>\n

                A few of our Most worthy knowledge isn\u2019t in a database \u2014 it\u2019s in unstructured textual content, comparable to buyer conversations. We not too long ago had 76 hours of Quarterly Enterprise Evaluate (QBR) recordings <\/strong>\u2014 a goldmine of shopper suggestions that was virtually unattainable to investigate manually.<\/p>\n

                The strategy: Bypassing the context window<\/h3>\n

                We used Cursor with Claude-4-Sonnet to construct an iterative ETL pipeline for textual content.<\/p>\n

                  \n
                1. Context definition:<\/strong> We outlined a immediate focusing on particular subjects (metrics, benchmarks, suggestions)<\/li>\n
                2. Software technology:<\/strong> Cursor generated a Python script to course of 116 transcript recordsdata<\/li>\n
                3. Iterative extraction:<\/strong> The script iterated by way of recordsdata, extracting related sentences into structured CSVs, which had been then summarized<\/li>\n<\/ol>\n

                  The impression<\/h3>\n

                  This strategy gave us a blended view of qualitative and quantitative insights: frequency counts of subjects alongside exemplar quotes.<\/p>\n

                  Extra importantly, it democratized the workflow. Vytautas Jak\u0161tys, our Product Director, a non-technical chief, now makes use of this similar technique. He generates SQL from our dbt docs utilizing Claude, then makes use of Cursor to investigate buyer chats to know the \u201cwhy\u201d behind the numbers.<\/p>\n

                  Last ideas on knowledge as a dialog<\/h2>\n

                  We aren\u2019t stapling AI onto our stack for present we’re baking it into how Omnisend asks, solutions, and acts.<\/p>\n

                  The result’s a division that ships fashions quicker, critiques code smarter, and lets everybody discover reliable knowledge and not using a guided tour. AI handles the mundane work \u2014 constructing new knowledge fashions from enterprise necessities, writing YAML documentation and checks, checking syntax and proper mannequin use, validating and reviewing, and discovering charts \u2014 clearing the runway for us to concentrate on the actual query:<\/p>\n

                  What\u2019s the following step that strikes us forward?<\/strong><\/p>\n

                  The following step is to proceed codifying our judgment into the markdown recordsdata: guidelines, pointers, kinds, and extra. It\u2019s an ever-evolving course of. As new LLM fashions emerge, so do new prompting strategies and approaches.\u00a0<\/p>\n

                  Most significantly, such workflows run totally on well-curated metadata. Your AI is just nearly as good as your documentation.<\/strong><\/p>\n

                  When you personal a dataset, undertake the model information and certify your belongings. You aren\u2019t simply serving to a human reader in the present day, you\u2019re making the assistant smarter for tomorrow.<\/p>\n<\/p><\/div>\n