Data mesh has emerged as a new paradigm for data management in recent years, allowing for inclusivity no matter the technology, should it be a data lake, data hub, data warehouse or S3 bucket.
The technologies are just nodes on the data mesh. Of course, at that point, it must be governed and people must self-serve. These days, multicloud is also a reality, which comes with various advantages.
“You put your data, your applications, and let’s pick on AWS for a second, U.S. East in AWS, which is the busiest area that they have, if it goes down, for my business to continue, I would probably want to move it to, say, Azure, hypothetically speaking, or Google,” Durgin said.
But the other parts that come to mind for Durgin are the companies, tools and applications being built in the cloud. That poses some interesting questions about how companies talk to each other and how data is shared.
“We’re getting to a world where every company is a data company. We all collect a lot of data. We want to use it for whatever that is,” Durgin said. “So how do we do it in a very smooth, easy way which opens additional opportunities for companies?”
Durgin discussed those questions and more with theCUBE industry analyst Dave Vellante at the Supercloud2 event, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. (* Disclosure below.)
On clouds and data
During the last year’s Snowflake Summit, Durgin discussed how Saks was modernizing its databases and pipelines to gain near real-time insights. Of course, Snowflake Inc. has put a big emphasis on data sharing, which is similarly a focus of Databricks Inc. and its open technology. But what are the trade-offs between openness and freedom versus control?
In Durgin’s view, it depends on the use case and the application. Companies would be limiting themselves, however, if they just stick to one cloud, especially given the applications starting to be built on top of clouds, she added.
“You have to govern and you have to control. And, honestly, it’s not a ‘nice-to-have’ anymore. There are legal compliances around data,” Durgin said. “I don’t like when my data’s used when I don’t know how. It’s a little creepy, right? So we have to come up with standards around that.”
Would Durgin like to one day see open standards actually emerge to enable that? And does she think that would be the “great disrupter” to the proprietary stack?
“I think for us to smoothly exchange data across various systems, various applications, we’ll have to agree to have standards,” she stated.
Head in the supercloud
One of the essential characteristics of the supercloud is the platform-as-a-service layer, which provides consistency across clouds and has unique attributes specific to the purpose of that supercloud. It could be data sharing for Snowflake, consistent infrastructure, or self-serve infrastructure for VMware.
“From a developer perspective, what do you hear from developers in terms of what they want? Are we close to getting that across clouds?” Vellante asked Durgin.
Developers always want freedom and the ability to engineer, according to Durgin.
“I always want to build something,” she said. “I think we’ll land there, but not out of the kindness of our own hearts. I think, as a necessity, we will have to agree to standards, and that’ll move the needle.”
So what limitations does Durgin see of cloud and cross-cloud, given that one cloud can’t do it all?
Durgin cited her background growing up in the data center, where things were physically limited by space. In such a space, you can only put so many racks in the data center before you run out of space. Even Earth has limited space, she noted.
“We’re not just data for the sake of collecting it anymore. We truly can’t take advantage of it, because servers have enough power to crank through it. We will run out of space,” Durgin said. “So how do we balance that? How do we balance that data across all the various data centers?”
Such concerns pose some big questions moving forward.
“I know I’m kind of talking crazy, but until we figure out how to build a data center on the moon, we will have to figure out how to take advantage of all the compute capacity that we have across the world,” she said.
Visit theCUBE’s Supercloud2 event page to watch full episodes on demand.
(* Disclosure: This is an editorial segment. While theCUBE is a paid media partner for Supercloud2, the sponsors for theCUBE’s event coverage do not have editorial control over content on theCUBE or SiliconANGLE.)