Talk
How Prefect unified Snorkel's orchestration stack
We use a diverse set of compute infrastructures for different jobs. We used to use redis queues to handle our delegation (operational nightmare). Now, we use prefect as the chief orchestrator, and have specialized workers for different tasks.
About
Software engineer with six years of experience working across the full stack. Particularly interested in the intersection of AI and infrastructure, with a focus on emerging technologies that accelerate existing workflows.