E40: Meta's MEGABYTE Revolution with Lili Yu of Meta AI

Nathan Labenz sits down with Lili Yu, a researcher of Meta AI to discuss the paper she authored: MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. In this conversation, they discuss the architecture and breakthroughs of their research, and the opportunity to eliminate the need for tokenization.

RECOMMENDED PODCAST:

Founding a business is just the tip of the iceberg; the real complexity comes with scaling it. On 1 to 1000, hosts Jack Altman and Erik Torenberg dig deep into the inevitable twists and turns operators encounter along the journey of turning an idea into a business. Hear all about the tactical challenges of scaling from the people that built up the world’s leading companies like Stripe, Ramp, and Lattice. Our first episode with Eric Glyman of Ramp is out now: https://link.chtbl.com/1to1000

The Cognitive Revolution is a part of the Turpentine podcast network. Learn more: www.turpentine.co

LINK:

MEGABYTE Paper: https://arxiv.org/pdf/2305.07185.pdf

TIMESTAMPS:

(00:00) Episode preview

(07:41) Takeaways from Lili Yu's paper: MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

(17:00) Architecture

(24:59) Embeddings

(27:43) Different local models

(34:23) Encoder model

(36:35) Transformer Architecture

(48:10) Choosing patch size

(01:08) What happens when you scale up?

(01:19:20) Big picture for Meta AI

(01:22:57) Responsible AI

(01:27:02) China and AI

TWITTER:

@labenz (Nathan)

@liliyu_lili (Lili)

@eriktorenberg (Erik)

@cogrev_podcast

SPONSOR:

Thank you Omneky (www.omneky.com) for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off.

MUSIC CREDIT:

MusicLM: