Indeed modernizes a petabyte-scale data lake using Amazon S3 Tables

Indeed’s mission is to help people get jobs. Behind every search result, every match recommendation, and every AI-powered insight, is a data lake that serves thousands of unique internal users who run 190,000 queries daily across teams spanning product development, marketing, sales, and data science. “Data is paramount to our mission, both directly and indirectly,” says Chris Voss, Director of Data Platform at Indeed. "Directly, we need high-quality data with lots of different signals for our matching algorithms. Indirectly, we need it for A/B testing, financial reporting, marketing analysis, and our sales organizations. We democratize access to data because we believe it has made a material difference in Indeed's success."

Indeed chose to switch from a hybrid cloud infrastructure to 100% hosting on Amazon Web Services (AWS) after a severe ice storm knocked out the company's Austin, Texas data center and threatened business continuity. "We went from 'we can't afford to migrate to the cloud' to 'if the Texas power grid went down for a month, we would be out of business,'" recalls David Schwartz, Engineering Manager at Indeed. "Suddenly, resiliency became a top priority for us." The company migrated its entire on-premises Hadoop infrastructure to AWS, including Amazon S3 for storage, Amazon EMR with Apache Spark for big data processing, AWS Glue Data Catalog for metadata management, and Amazon Athena for serverless querying. An initial lift-and-shift gave Indeed the redundancy, scalability, and peace of mind it needed to grow without the limitations of physical hardware.  

By mid-2023, the migration was complete, but Indeed's data needs had already outgrown the legacy Hive/ORC format it carried into the cloud. Data science teams were rewriting years of data multiple times daily because Hive's write-once model required full partition rewrites for even single-row updates. Queries against tables with thousands of partitions took hours. Schema changes required dropping entire tables and backfilling from scratch.  Moving data to Apache Iceberg on Amazon S3 delivered schema evolution, partition pruning, multi-engine interoperability, and row-level updates that transformed expensive full-table rewrites into lightweight snapshot operations. S3 provided unmatched durability, availability, and the ability to store virtually any amount of data.

Indeed began running Iceberg on Amazon S3 in late 2023, but operating the format at scale proved more difficult than adopting it. The team built a custom maintenance application using Spark jobs on an Amazon EMR cluster to handle compaction, snapshot expiration, and orphan file deletion. The application ran 200,000 Spark job executions monthly, consumed over 1,000 developer-hours annually in upkeep, and still could not keep pace with high-velocity writers. "Getting maintenance that is not just cost-effective, but fast enough to keep up with some systems is a hard engineering task," says Schwartz. "We got to the point where our own Iceberg maintenance was taking too much time. As a manager, I would much rather rely on AWS to solve that challenge."

When AWS introduced Amazon S3 Tables, Indeed's team recognized the potential immediately. S3 Tables are purpose-built for Apache Iceberg storage, providing automated compaction, snapshot management, and unreferenced file cleanup as fully managed capabilities. After conducting a cost-benefit analysis, Indeed chose Amazon S3 Tables over self-managed Iceberg for three reasons: automated maintenance that eliminated the need for custom Spark infrastructure, table-level resource policies that replaced per-object tagging across 530 million objects, and a built-in 10x increase in transaction throughput that resolved rate-limiting incidents that had caused multiple high-severity outages. "The table resource policy is a game changer," says Venkatesh, Tech Lead, Data Lake Team at Indeed. "Previously, we tagged each of our 530 million S3 objects individually during ingestion. Now, we set one policy on the table. If a policy changes, we update it in microseconds instead of re-tagging millions of objects which in some instances had run for 10+ hours for one table."

Indeed’s migration from self-managed Iceberg to S3 Tables uses a dual-catalog linking strategy that connects the S3 Tables catalog to Indeed's existing AWS Glue Data Catalog. This approach maintains full compatibility with the company's four query engines, including Athena, while enabling a phased cutover that keeps daily query volume running without interruption. 

Indeed also appreciates that S3 Tables offer S3 Intelligent-Tiering and automatic replication, which replaced a complex backup strategy that stored every partition rewrite in Amazon S3 Glacier Deep Archive. The new approach maintains 30 days of non-current snapshots in a live replica, enabling point-in-time recovery without the cost and complexity of batch replication jobs.

Indeed's migration to S3 Tables is already delivering measurable results. The company expects to eliminate over 1,000 annual developer hours previously consumed by Iceberg maintenance, reclaim four developer months for product work, and achieve a 10 percent cost savings over its existing data lake by removing the compute overhead of self-managed compaction jobs. Indeed also plans to simplify their data pipelines by writing directly to S3 Tables, eliminating duplicate data copies and reducing the cost of replication. Data onboarding for new producer teams is expected to drop from a full day of manual setup to minutes. Security reclassification, which previously required re-tagging millions of objects over weeks, will now take a single policy update across 18,000 datasets.

"Having a managed solution that is effectively the expert at managing Iceberg, versus us having to have that expertise, maintain that expertise, evolve that expertise as the ecosystem evolves, the advantages are clear," says Schwartz. 

This data modernization is positioning Indeed for what comes next. The company has built an internal natural language data querying tool powered by AI and is beginning to leverage  semantic modeling to make its data lake accessible to AI agents. Iceberg's role as a first-class format for semantic models and its interoperability across engines means that every dataset migrated to S3 Tables becomes immediately available to these new AI-driven workflows. "As we get Iceberg and the REST catalog in place, our customers will have access to all of the data in all of the places they want it," says Voss. "I want to see how the business turns that into better results for job seekers and employers."

For Indeed, the journey from an on-premises Hadoop cluster in Austin to a managed, AI-ready data lake on AWS is more than a technical migration. It is the infrastructure that makes it possible to help people get jobs faster, more accurately, and at a scale that continues to grow.