Meta on 19 October detailed Prometheus, the gigawatt-scale supercluster at its New Albany, Ohio, data centre. The company says its completed first phase trained the Muse family of foundational models, including Muse Wonder, and that it is building towards roughly 1 million GPUs. The disclosure puts hardware and operating figures behind models that AI Affairs previously reported Meta had released.
Key points
- Meta says Prometheus already contains hundreds of thousands of GPUs across interconnected buildings.
- Its completed first phase was used to train Muse models, including Muse Wonder.
- Meta reports more than 85% goodput on Muse Wonder pre-training runs and more than 90% on reinforcement learning runs.
- Two Nvidia NVL72 platforms are deployed, with hardware designed to give each GPU 800G of network capacity beyond its rack.
New Albany moves from web traffic to Muse
Two years ago, Meta says, the New Albany site was handling conventional web traffic. It reworked the data centre for model training and is expanding it in phases, using conventional buildings, buildings designed for AI and rapidly deployed structures linked by fibre. Meta says Prometheus now contains hundreds of thousands of GPUs spread across multiple buildings. The planned roughly 1 million GPUs would be distributed, but the company intends them to work together as one training system.
That distinction matters because a model-training job can involve processors in more than one building, rather than filling a single room with machines. Meta says the capacity lets it pursue larger models and longer context windows — the amount of material a model can take into account during a task — than it previously found practical. The completed phase was used for Muse, while the company says it is continuing to expand Prometheus with further models in mind.
Catalina and Clemente put 72 GPUs together
Meta currently uses two Nvidia NVL72 platforms at Prometheus. In each, 72 GPUs can work closely together through NVLink, the connection between processors within the same hardware group. Its Catalina platform uses GB200 NVL72 in a two-rack arrangement, with 36 GPUs in each rack. Meta lists 13.4 TB of HBM3e memory and 720 peak PFLOPS of FP8 training compute for the pair. Those are hardware specifications, rather than measurements of a completed Muse training run.
The processors also need to exchange information beyond that close-knit group. Meta says it chose a Catalina configuration with two ConnectX-7 400G network cards for each GPU, providing 800G of capacity per GPU for those wider connections. Its engineers expected a narrower connection to hold back a GB200 system whose compute power, memory bandwidth and within-group connections had all grown relative to the H100. The choice puts network capacity alongside processor specifications as part of the training design.
Clemente, the second deployed platform, uses GB300 NVL72. Meta says it increases memory per GPU from 192 GB to 288 GB, while ConnectX-8 SuperNICs provide 800G per GPU without Catalina’s two-card arrangement. A different GPU-to-CPU ratio lets Meta fit Clemente’s full 72-GPU group into one rack, rather than Catalina’s two. Meta currently has two GPU generations deployed at Prometheus.
Prometheus links training across buildings
The links between those groups form another part of the system. StorageReview reported that Prometheus has a dedicated Ethernet network for training, separate from the network carrying ordinary service traffic. Each building’s training network can connect up to 20,736 GPUs, with an aggregation network joining buildings. The arrangement keeps the most time-sensitive exchanges within a building and carries other training traffic across the wider site.
A training job sends information back and forth repeatedly, much as a group working on one document must keep passing updated pages around. Keeping the busiest exchanges nearby reduces the distance those messages travel. The wider network described by StorageReview spans links from 2km to more than 100km, and Meta worked with its model-training team on the delays involved in running pre-training across those distances.
Writing a long document could involve keeping more of its earlier pages in view as the work develops, if the longer context windows Meta says Prometheus makes practical become available in a model used for that task. The same network design would matter during training because the processors building that model have to keep exchanging information.
Muse Wonder runs and 47,000 GPU hours
Meta reports more than 85% goodput for Muse Wonder pre-training runs and more than 90% for its reinforcement learning runs. Goodput concerns productive training time: as Meta describes it, a fault on one host can slow or stop the whole job even while the other hardware remains available. It is the difference between having machines ready to work and keeping the shared job moving.
During those runs, Meta measured an overall wall-clock mean time between interruptions of approximately 47,000 GPU hours. GPU hours add up operating time across processors, so the unit should not be read as 47,000 hours passing on a clock between two stops. The company says subsequent changes across its fleet, including firmware fixes, raised the measure to more than 54,000 GPU hours.
Meta’s stated reliability target is two interruptions per 8,000 GPUs per day, StorageReview reported. The publication put that target at roughly 100,000 GPU hours between interruptions.
