I work in data warehousing. When you strip away all the fluff and jargon, a data warehouse is just a set of materialized views. Our work consists of taking data from tables and storing the same data in tables but in a different shape.
The problem of efficiently updating materialized views is as old as the 1986 paper of the same name. The problem is simple to state: A change in a source system table must be converted to a set of changes in the materialized view tables. 30+ years and hundreds of papers later, a general-purpose relational solution is nowhere in sight. Data warehouses still rely on an army of administrators using point-and-click tools to move data in. I think we can confidently say that the relational model is just not up to the task.
Intuitively, I feel that the problem of updating materialized views is solvable with Terminus because Terminus’ query language can be represented as data itself and reasoned about. I’m currently trying to show it, but it will take me a while.
I would appreciate any thoughts or feedback on this topic.
I think Terminus is uniquely positioned to solve this problem once and for all. There’s quite a bit of money on the line too. I would estimate hundreds of millions of dollars are spent annually on point-and-click ETL tools like Informatica PowerCenter. Every sane data warehouse architect hates these tools and knows there must be a better solution…