Foundational Data Engineering At Two Sigma
Summary
In this episode of the Data Engineering Podcast Effie Baram, a leader in foundational data engineering at Two Sigma, talks about the complexities and innovations in data engineering within the finance sector. She discusses the critical role of data at Two Sigma, balancing data quality with delivery speed, and the socio-technical challenges of building a foundational data platform that supports research and operational needs while maintaining regulatory compliance and data quality. Effie also shares insights into treating data as code, leveraging modern data warehouses, and the evolving role of data engineers in a rapidly changing technological landscape.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
This episode is brought to you by Coresignal, your go-to source for high-quality public web data to power best-in-class AI products. Instead of spending time collecting, cleaning, and enriching data in-house, use ready-made multi-source B2B data that can be smoothly integrated into your systems via APIs or as datasets. With over 3 billion data records from 15+ online sources, Coresignal delivers high-quality data on companies, employees, and jobs. It is powering decision-making for more than 700 companies across AI, investment, HR tech, sales tech, and market intelligence industries. A founding member of the Ethical Web Data Collection Initiative, Coresignal stands out not only for its data quality but also for its commitment to responsible data collection practices. Recognized as the top data provider by Datarade for two consecutive years, Coresignal is the go-to partner for those who need fresh, accurate, and ethically sourced B2B data at scale. Discover how Coresignal's data can enhance your AI platforms. Visit dataengineeringpodcast.com/coresignal to start your free 14-day trial.
Your host is Tobias Macey and today I'm interviewing Effie Baram about data engineering in the finance sector
Interview
Introduction
How did you get involved in the area of data management?
Can you start by outlining the role of data in the context of Two Sigma?
What are some of the key characteristics of the types of data sources that you work with?
Your role is leading "foundational data engineering" at Two Sigma. Can you unpack that title and how it shapes the ways that you think about what you build?How does the concept of "foundational data" influence the ways that the business thinks about the organizational patterns around data?
Given the regulatory environment around finance, how does that impact the ways that you think about the "what" and "how" of the data that you deliver to data consumers?
Being the foundational team for data use at Two Sigma, how have you approached the design and architecture of your technical systems?How do you think about the boundaries between your responsibilities and the rest of the organization?
What are the design patterns that you have found most helpful in empowering data consumers to build on top of your work?
What are some of the elements of sociotechnical friction that have been most challenging to address?
What are the most interesting, innovative, or unexpected ways that you have seen the ideas around "foundational data" applied in your organization?
What are the most interesting, unexpected, or challenging lessons that you have learned while working with financial data?
When is a foundational data team the wrong approach?
What do you have planned for the future of your platform design?
Contact Info
LinkedIn
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
Links
2Sigma
Reliability Engineering
SLA == Service-Level Agreement
Airflow
Parquet File Format
BigQuery
Snowflake
dbt
Gemini Assist
MCP == Model Context Protocol
dtrace
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
LLMs for Equities Feature Forecasting at Two Sigma with Ben Wellington - #736
Today, we're joined by Ben Wellington, deputy head of feature forecasting at Two Sigma. We dig into the team’s end-to-end approach to leveraging AI in equities feature forecasting, covering how they identify and create features, collect and quantify historical data, and build predictive models to forecast market behavior and asset prices for trading and investment. We explore the firm's platform-centric approach to managing an extensive portfolio of features and models, the impact of multimodal LLMs on accelerating the process of extracting novel features, the importance of strict data timestamping to prevent temporal leakage, and the way they consider build vs. buy decisions in a rapidly evolving landscape. Lastly, Ben also shares insights on leveraging open-source models and the future of agentic AI in quantitative finance.
The complete show notes for this episode can be found at https://twimlai.com/go/736.
The things engineers are desperate for PMs to understand | Camille Fournier (author of “The Manager’s Path,” ex-CTO at Rent the Runway)
Camille Fournier is the author of The Manager’s Path, which many consider the definitive guide for navigating one’s career path in tech. Camille was previously the CTO of Rent the Runway, VP of Technology at Goldman Sachs, Head of Platform Engineering at Two Sigma, and Global Head of Engineering and Architecture at JPMorgan Chase. She is about to release new newest book, Platform Engineering: A Guide for Technical, Product, and People Leaders. In our conversation, we discuss:
• What product managers do that annoys engineers
• Why major rewrites are a trap
• Why you should have fewer one-on-ones
• Strategies for organizing and working with platform teams
• Tips for new managers
• Advice for transitioning from individual contributor to manager
• Much more
—
Brought to you by:
• DX—A platform for measuring and improving developer productivity
• CommandBar—AI-powered user assistance for modern products and impatient users
• Coda—The all-in-one collaborative workspace
—
Find the transcript and show notes at: https://www.lennysnewsletter.com/p/engineering-leadership-camille-fournier
—
Where to find Camille Fournier:
• LinkedIn: https://www.linkedin.com/in/camille-fournier-9011812/
• Website: https://skamille.medium.com/
—
Where to find Lenny:
• Newsletter: https://www.lennysnewsletter.com
• X: https://twitter.com/lennysan
• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/
—
In this episode, we cover:
(00:00) Camille’s background
(02:17) Common annoyances between PMs and engineers
(07:09) Avoiding the telephone game
(08:05) Hoarding ideas and over-engineering
(09:55) The importance of involving engineers in ideation
(11:37) The middle-person dilemma
(14:21) Rewriting systems: a big trap?
(20:40) Engineering leadership lessons
(36:02) Moving from IC to management
(40:32) One-on-one meetings
(45:10) Pushing beyond comfort zones
(45:27) Building a balanced work culture
(48:01) Effective time management strategies
(54:15) Advice for platform team success
(01:02:42) Platform team responsibilities
(01:04:43) When to form a platform team
(01:07:02) Thriving on a platform team
(01:12:48) AI corner
(01:17:03) Lightning round and final thoughts
—
Referenced:
• Platform Engineering: A Guide for Technical, Product, and People Leaders: https://www.amazon.com/Platform-Engineering-Technical-Product-Leaders/dp/1098153642/
• The Manager’s Path: A Guide for Tech Leaders Navigating Growth and Change: https://www.amazon.com/Managers-Path-Leaders-Navigating-Growth/dp/1491973897
• 97 Things Every Engineering Manager Should Know: Collective Wisdom from the Experts: https://www.amazon.com/Things-Every-Engineering-Manager-Should/dp/1492050903
• Avoiding the Rewrite Trap: https://skamille.medium.com/avoiding-the-rewrite-trap-b1283b8dd39e
• Levelsio on X: https://x.com/levelsio
• Pieter Levels on the Lex Fridman Podcast: https://www.youtube.com/watch?v=oFtjKbXKqbg
• GraphQL: https://graphql.org/
• New Blue Sun by André 3000 on Spotify: https://open.spotify.com/album/33Ek6daAL3oXyQIV1uoItD
• Musk’s 5 Steps to Cut Internal Bureaucracy at Tesla and SpaceX: https://icecreates.com/insight/musk-s-5-steps-to-cut-internal-bureaucracy-at-tesla-and-spacex-you-may-say-it-s-his-algorithm/
• Ian Nowland on LinkedIn: https://www.linkedin.com/in/inowland/
• Studio Pulls ‘Megalopolis’ Trailer Using Fake Quotes from Famed Movie Critics: https://www.huffpost.com/entry/studio-pulls-megalopolis-trailer-using-fake-quotes-from-famed-movie-critics_n_66c74046e4b0f1ca469413c7
• Claude 2: https://www.anthropic.com/news/claude-2
• What Got You Here Won’t Get You There: How Successful People Become Even More Successful: https://www.amazon.com/What-Got-Here-Wont-There/dp/1401301304
• When Things Fall Apart: Heart Advice for Difficult Times: https://www.amazon.com/When-Things-Fall-Apart-Difficult/dp/1611803438
• Alien: Romulus: https://www.imdb.com/title/tt18412256/
• Whoop: https://www.whoop.com
—
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.
—
Lenny may be an investor in the companies discussed.
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.lennysnewsletter.com/subscribe
794: Exciting (and Frightening!) Trends in Open-Source AI
Trends in open-source AI: Join Jon Krohn and a panel of data science icons as they discuss the most exciting and concerning developments in open-source AI. Hear insights from Drew Conway, Jared Lander, Emily Zabor, and JD Long on the transformative potential of AI and its future impact.
Additional materials: www.superdatascience.com/794
Interested in sponsoring a SuperDataScience Podcast episode? Email natalie@superdatascience.com for sponsorship information.
790: Open-Source Libraries for Data Science at the New York R Conference
The experts reveal their top open-source R libraries with us live from the New York R Conference! This Super Data Science Podcast episode features an exclusive panel with data science trailblazers Drew Conway, Jared Lander, Emily Zabor, and JD Long. They share their favorite R libraries and valuable insights to enhance your data science practice.
Additional materials: www.superdatascience.com/790
Interested in sponsoring a SuperDataScience Podcast episode? Visit passionfroot.me/superdatascience for sponsorship information.
511: Data Science for Private Investing — LIVE with Drew Conway
Drew Conway joins us on the first live podcast to discuss his work in private investing and how data science figures into and improves his work.
In this episode you will learn:
The R Conference and NYHackR [6:33]
Machine Learning for Hackers [20:17]
Two Sigma and Drew’s work [28:27]
Drew’s team structure at Two Sigma [35:12]
Audience Q&A [46:27]
Additional materials: www.superdatascience.com/511
Mike Schuster on whether AI can help hedge fund investors to beat the market?
Episode ten of The Robot Brains Podcast investigates how hedge funds are using AI to find competitive advantages for their investments. Joining us to explain how AI is starting to be used for pure profit is Mike Schuster Managing Director and Head of AI Core Team at the New York-based Financial Sciences firm: Two Sigma. With decades of experience working in AI (including Google where he pioneered the ML technology that became Google Translation), when Mike first joined Two Sigma he was described as potentially being the “hedge fund’s last human employee”. In his chat with our host Pieter Abbeel, Mike explains that it actually takes a team to bring AI to banking, as well as explaining how working for a hedge fund is like working for a tech company and the future potential of AI robot investors.
Host: Pieter Abbeel | Executive Producers: Ricardo Reyes & Henry Tobias Jones | Audio Production: Kieron Matthew Banerji | Title Music: Alejandro Del Pozo
Hosted on Acast. See acast.com/privacy for more information.
437: Data Science at a World-Leading Hedge Fund
Claudia Perlich joins us to discuss her work at one of the world’s largest hedge funds and how she got to work there, as well as her history of winning data science competitions.
In this episode you will learn:
Life and work during the pandemic [2:23]
Claudia’s history with horses and riding [8:28]
Claudia’s work at Two Sigma [12:00]
Claudia’s role on a daily basis [20:51]
Tools of the trade [30:27]
What Claudia looks for when hiring [36:37]
What skills do future hires need? [40:32]
Claudia’s history with data science competitions [48:22]
Why work in finance and at Two Sigma? [1:00:19]
Additional materials: www.superdatascience.com/437
Camille Fournier on Building Tech at Two Sigma
We talk a lot about quantitative trading on the podcast, but typically from a rather big picture perspective, and not at the level of actually building the systems needed for trading and data analysis. On this episode, we speak with Camille Fournier, the head of Platform Engineering at Two Sigma, the financial services firm that, among other things, runs a large hedge fund. Fournier, previously the CTO at Rent the Runway, discusses how her job works, the challenge of managing software engineers, and how tech within a financial services company is different than tech within a consumer-facing startup.
See omnystudio.com/listener for privacy information.
Supporting Rapid Model Development at Two Sigma with Matt Adereth & Scott Clark - TWIML Talk #273
Today we’re joined by Matt Adereth, managing director of investments at Two Sigma, and return guest Scott Clark, co-founder and CEO of SigOpt, to discuss:
• The end to end modeling platform at Two Sigma, who it serves, and challenges faced in production and modeling.
• How Two Sigma has attacked the experimentation challenge with their platform.
• What motivates companies that aren’t already heavily invested in platforms, optimization or automation, to do so, and much more!
Why One Of The Most Successful Quant Funds Decided To Create Its Own Video Game
Quantitative finance is red hot. These days, basically everyone (banks, hedge funds etc.) is hiring mathematicians and coders. So what differentiates one quant shop from any other? On this week's episode of the Odd Lots podcast, we speak to Alfred Spector, the CTO of Two Sigma Investments, which is one of the most successful quant firms in the world. Spector is a computer scientist who previously did long stints at both Google and IBM. He tells us about why Two Sigma spent resources to create its own video game, and what the firm does to ensure that technologists and mathematicians are eager to work there.
See omnystudio.com/listener for privacy information.
Cognitive Biases in Data Science with Drew Conway - TWiML Talk #39
This show features my interview with Drew Conway, whose Wrangle keynote could have been called “Confessions of a CIA Data Scientist.” The focus of our interview, and of Drew’s presentation, is an interesting set of observations he makes about the role of cognitive biases in data science. If your work involves making decisions or influencing behavior based on data-driven analysis--and it probably does or will--you’re going to want to hear what he has to say. A quick note before we dive in: As is the case with my other field recordings, there’s a bit of unavoidable background noise in this interview. Sorry about that! The show notes for this episode can be found at https://twimlai.com/talk/39