🤯 AI's Secret: Decoding Human Action 🧠

THE RISE OF PHYSICAL AI TRAINING DATA
Encord, operating from a warehouse in San Leandro, California, is pioneering a novel approach to training artificial intelligence models for robotics. The company’s core business revolves around manufacturing training data, recognizing a critical constraint in the development of humanoid and warehouse robots – the scarcity of real-world physical training data.

BRAIN-WAVE DATA: A NEW DIMENSION
Zander Labs, a German neuroscience startup, is collaborating with Encord to develop a system that measures brainwave activity during robot training tasks. This data, intended to capture mental states like error, intent, and surprise, aims to provide a more nuanced and useful dataset for training AI models. The initial trial involves Andrew Ceja, Encord’s “pilot,” meticulously disassembling a robotic block tower while wearing a brainwave headset.

ECCEGONIC VIDEO & ROBOTIC PAIRS
Encord’s data collection strategy heavily relies on “egocentric” video – footage captured by workers wearing cameras – often augmented with additional camera angles and metrics. They utilize leader-follower rigs, pairing robotic arms controlled by human operators with robotic arms that mimic their movements, to generate data for tasks such as pouring coffee and stacking poker chips. This approach mirrors the data collection methods used by self-driving car companies, though on a smaller, more targeted scale.

DATA MODALITIES: EXPANDING THE SENSORY INPUT
Beyond video, Encord is exploring other data modalities, including sensors strapped to forearms to detect muscle electrical signals and 3D depictions of hand movements based on arm sensor data. This aims to overcome the limitations of traditional video data, which often fails to capture the full dexterity of the human hand. The data is meticulously annotated with physical descriptions, such as "right hand tightens bolt," to aid LLM-based models.

THE ECONOMICS OF PHYSICAL DATA GENERATION
The creation of physical training data represents a significant shift in the robotics industry. Unlike scraping text from the internet, which was a near-costless process for LLM developers, generating physical data requires substantial investment. Encord’s approach acknowledges this reality, aiming to provide a more efficient and targeted data solution.

A NETWORKED APPROACH TO TRAINING
Encord’s work extends beyond simply creating data sets; it’s building a network of knowledge and expertise. The company collaborates with numerous robotics firms, leveraging its insights to identify emerging data techniques and trends. This vantage point, coupled with its pilot program, positions Encord as a key player in shaping the future of physical AI.

THE HUMAN ELEMENT: PILOTS & EXPERTISE
The Encord facility employs a team of “pilots,” like Andrew Ceja and previously Scale employees, who possess diverse backgrounds, including experience at OpenAI’s robot lab and Berkshire Grey. Ceja’s previous role at a waste management company, overseeing a robotic trash sorter, highlights his practical understanding of robotics challenges and his enthusiasm for solving training tasks.

SPECIFIC TASKS & DATA GENERATION
Encord’s pilots are currently engaged in generating data for a wide range of tasks, including manipulating ethernet cables, plugging and unplugging devices, and handling various objects like fake flowers, books, and kitty litter trays. This diverse set of activities demonstrates the company's commitment to creating a comprehensive and adaptable training dataset.

LOOKING AHEAD: CONTINUED INNOVATION
Encord continues to innovate in its approach to physical AI training data, constantly refining its methodologies and exploring new data modalities. The company’s ongoing research and development efforts, combined with its network of collaborations, are poised to play a crucial role in the advancement of robotics and artificial intelligence.