“AI has been the wild west”: Creating standards for agents with Sean Falconer
Confluent is an all-in-one, real-time platform that allows you to stream, connect, process, and govern your data.
Connect with Sean on Linkedin, and listen to the Software Engineering Daily podcast, where he co-hosts.
Do you know the answer to Loadrunner lr_get_attrib_string always returns null? If so, you can help out Trncvs and receive the bounty offered on the question.
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
From startup to Google and back again
Sean hosts Partially Redacted, a podcast about data privacy, security, and compliance.
He also hosts the podcast Software Engineering Daily, which features technical interviews on everything from the ethics of GPTs to cloud-native search and WebAssembly. Start with the recent episode Surviving ChatGPT with Christian Hubicki (of Survivor fame).
You can also read about how he crowdsourced a behavioral model for Survivor.
Sean spent four years working in developer relations (DevRel) at Google. Here’s a Software Engineering Daily episode about the role DevRel plays at Google.
Connect with Sean on LinkedIn or Twitter (I mean, X), or check out his website.
Kudos to Great Question badge winner Kai Sellgren for asking How to remove an element from a vector given the element?.
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault
Summary
The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems. In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.
Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. By the time errors have made their way into production, it’s often too late and damage is done. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. No more shipping and praying, you can now know exactly what will change in your database! Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.
Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
Your host is Tobias Macey and today I’m interviewing Sean Falconer about the idea of a data privacy vault and how the Skyflow team are working to make it turn-key
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what Skyflow is and the story behind it?
What is a "data privacy vault" and how does it differ from strategies such as privacy engineering or existing data governance patterns?
What are the primary use cases and capabilities that you are focused on solving for with Skyflow?
Who is the target customer for Skyflow (e.g. how does it enter an organization)?
How is the Skyflow platform architected?
How have the design and goals of the system changed or evolved over time?
Can you describe the process of integrating with Skyflow at the application level?
For organizations that are building analytical capabilities on top of the data managed in their applications, what are the interactions with Skyflow at each of the stages in the data lifecycle?
One of the perennial problems with distributed systems is the challenge of joining data across machine boundaries. How do you mitigate that problem?
On your website there are different "vaults" advertised in the form of healthcare, fintech, and PII. What are the different requirements across each of those problem domains?
What are the commonalities?
As a relatively new company in an emerging product category, what are some of the customer education challenges that you are facing?
What are the most interesting, innovative, or unexpected ways that you have seen Skyflow used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Skyflow?
When is Skyflow the wrong choice?
What do you have planned for the future of Skyflow?
Contact Info
LinkedIn
@seanfalconer on Twitter
Website
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Skyflow
Privacy Engineering
Data Governance
Homomorphic Encryption
Polymorphic Encryption
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast