By QuickBooks and TurboTax of Intuit are two of the most popular consumer finance products ever produced. Both are trademarks of Intuit. However, Intuit is forging ahead with aspirations to transform itself into an AI juggernaut. Thanks to recent advancements in machine learning and the purchases of Credit Karma and Mailchimp. With more than 100 million users and close to $10 billion in revenue, Intuit has enterprise-scale needs for data processing.
Developing separate architectures for each big data project would have made the data silo issue worse. Intuit adopted a unified strategy that relies on a lakehouse as the enterprise-wide standard for data. Rajat Khare Intuit at #GIDS Live 2020 spoke on Automated Failure Injection and Testing across Microservices in a video. Alon Amit, the company’s vice president of product management, stated at the Databricks’ Data + AI Summit a few weeks ago that “Intuit is evolving into an AI-driven expert platform.” “We’re here to help people flourish, and that used to mean helping you file your taxes and balance your finances. But it has a lot more meaning now.
When Amit and Manish Amde, who is currently Intuit’s director of engineering, joined the business three years ago. The architecture wasn’t even thought about. Both Amit and Amde worked at Origami Logic, which Intuit bought to aid in the development of their data and AI infrastructure.
Unfolding Data
The new data architecture had to meet a number of criteria laid out by the data leaders. In order to promote an experimental culture, internal users first needed to be able to quickly develop data pipelines and receive results to queries. Additionally, they wanted a storage repository that could handle transactions. Intuit most urgently required a single data architecture that could provide data for many use cases.
Like most successful, $10 billion firms, Intuit had a certain amount of technology baggage when Amit and Amde first started working there. To find a solution, Amit and Amde would have to work within these limitations (and others). From their time at Oragami Logic, both were already familiar with Databricks. They both were aware of the capabilities of the platform. In the past, when Spark was still a largely unheard-of computing initiative at the UC Berkeley AMPLab, Amde had also collaborated with the Databricks founders.
The Rollout
Databricks’ Delta Lake was chosen by Intuit as the foundation for its new data architecture. Databricks claimed to have discovered a happy medium. It was between unmanageable data swamps and slow-to-adapt data warehouses by combining elements traditionally associated with a data warehouse. This involves (such as ACID transactions and quality guarantees) the scalability and flexibility benefits of a data lake. A data map is an important component of Intuit’s strategy. The physical layer, the operational layer, and the business layer are the three categories of data that make up the data map.
We can’t expect people to do it twice—once for the analyst and once for the data scientist—as Amit and Amde stressed how difficult it is to persuade the company’s culture to start producing data at scale with logic and business meaning. “Everyone is looking at the same data, therefore it’s better,”.
