In 2019, when I wrote The Slayers of Layers, I was prepared to leave the fate of layers to Charles Darwin. Given enough time, I figured, organisations with fewer layers to maintain would have an advantage over those with more. It seems like Darwinism may have failed us this time, given that I see things like the medallion architecture propped even by vendors.
Consider the data warehouse. It has two fixed points. At one end there is the landing zone, where data from the various source systems arrives. At the other there is the business-facing model, against which people ask their questions and from which recurring reports fetch their figures. Input and Output. The people delivering the data have an interest in the first. The people using the data have an interest in the second. Who, then, has an interest in all the things we have put in between?
We do, of course. The medallion layers of bronze, silver and gold are there for the benefit of the people building and maintaining the warehouse. There was some sense in this, even if I have been a strong proponent of slaying as many layers as possible. A developer who had seen one load could find their way around the next, and when something went wrong at two in the morning, familiarity had its merits. Still, there is a certain irony in building more things to maintain in the interest of making maintenance easier. It has at the very least kept us busy though.
Those days are now gone, or at least should be gone. An AI agent, for now boosted by a data warehousing skill with some best practices, can easily take data from the landing zone directly into the model when instructed. You want it running on a clock. No problem, the agent will set up the tasks for you in a loading graph you can schedule. What would have taken a developer days, and what would have become a structured but generic flow through the layers, now takes minutes and is specifically optimized for the purpose at hand. Did a source change dramatically overnight? Give the agent a few minutes to adapt next morning.
Surely, though, we still need a presentation layer? Who in their right mind would ask a business user to navigate a normalised and historised warehouse? Well, let us see what navigating it actually entails. It has richer representation than what can normally be found in the data marts, so the data marts are essentially a dumbed down version to make querying simpler. In order to use that rich pool of data, more complex queries are needed, but what may be complex for a business user is a breeze for an AI agent, especially if boosted with a semantic model. Subtleties like taking into account what we knew when and what was in effect are handled flawlessly by the agent.
What remains is the landing zone, a proper Inmon-style data warehouse model, and an AI agent. For the immediate future, the agents also need our help with skills and semantic models, but at some point we’re going to be slowing them down by being humans in the loop. Somebody also has to understand why the actuary and the accountant are both right, and these things strike me as a far better use of our time than shuffling data between superfluous layers.
The layers have been slain.
Someone is now likely thinking, can’t we skip the Inmon-style data warehouse modeled layer as well, and have agents use raw files from the landing zone directly? If you don’t immediately know the answer, I’ll leave it up to you to ask your favourite agent why that is not a good idea at all.