Immerse yourself in virtual worlds and see the future today

Ever wanted to attend a small, focused event where the UN and international representatives discuss how the latest technology will change life for most people on earth? You can. I’m going, and you’re welcome to come, too, to the third UN Virtual Worlds Day on 11-12 May.

Ambassadors, ministers, city leaders, industry pioneers, and academic leaders are talking about turning AI, spatial intelligence, and the Citiverse ecosystem into trusted, people-centred outcomes for cities and communities worldwide. It’s happening on in May, in Geneva. You can see the agenda, and register to come in person or online at UN Virtual Worlds.

From Gaussian splats to digital twins

I’ve been working in smart cities for a decade. Over that time, what were once just visions and dreams have incrementally come to pass: the Internet of Things (IoT) is now embedded worldwide, enabling real-time sensing and programmability (it wasn’t here a decade ago); geographic information systems (GIS) have gone through generational upgrades, enabling unprecedented levels of interactive vanilla and exotic 3D tiles; XR visualisation, and now Gaussian splats enable immersive and interactive place-based experiences, from anywhere (it wasn’t here a decade ago). 

Digital twins are being used in all sorts of urban situations from the fairly passive monitoring of the effectiveness of a housing retrofit, to high-intensity passenger management in mass-transit hubs – automating the opening and closing of ticket turnstiles based on passenger flow (definitely not here a decade ago). The early stages of autonomous systems, including humanoid robotics are burgeoning, especially in Asia, with lots more to come.

A decade ago this kind of progress was simply inspiring, today, we’re growing a bit more concerned about how this technology will be used, who’s in charge, and its environmental impacts. We’ve watched the movies, we’ve seen what’s possible, and increasingly we’re increasingly cautious. Now is the time for us to work together. Yes, we want to embrace the benefits of digital innovation, we need to make sure it nurtures natural ecosystems, vulnerable citizens, and we must ensure it produces joy, creativity, freedom, and care, alongside preserving public safety, security, efficiency.

A decade ago this kind of progress was simply inspiring, today, we’re growing a bit more concerned about how this technology will be used, who’s in charge, and its environmental impacts

The whole event in Geneva looks fascinating, but one session particularly caught my eye: Spatial Intelligence for Decision-Making in the Citiverse: Encoding Space, Behaviour, and Policy. The session’s description reads: “Spatial intelligence sits at the intersection of virtual worlds, AI, and the physical environment, enabling cities and institutions to integrate geospatial data, simulations, and AI-driven analytics to model real-world systems from urban planning and mobility to climate resilience and disaster response….”

Earlier this year, I started following the work of two leaders of world model start-ups, Yann LeCun, and Fei-Fei Li. Yann LeCun was formerly chief scientist at Meta Platform’s AI division, and is now the founder of AMI Labs, which recently raised more than $1bn to develop “world models”, foundation models that learn from reality, with LeCun aiming to enable AI to reason and understand physical laws. Fei-Fei Li ‘godmother of AI’, co-founded World Labs along with Justin Johnson, Christoph Lassner, and Ben Mildenhall, focusing on spatial intelligence – creating 3D, persistent, and interactive worlds from 2D images or text, with computer-vision and video models, being a major source of training-data for this approach.

Large language models and physical reality

By now we are familiar with large language models (LLMs), with their training data being the vast amount of text on the Internet. LLMs predict and produce words. At this point, they astonish us with their mastery of language, as though intelligent. But these LLMs do not, and cannot, comprehend our physical reality.

Every infant develops its parietal lobe (the sensing part of its brain) to learn that it hurts when you fall downstairs, try to walk through a wall, or put your hand in a flame. Protective parents intuitively know that it takes a while to develop, especially when infant mobility begins. Our parietal lobes interpolate our senses, vision, smell, touch, hearing, they literally make sense of it.

Our parietal lobes are doing well by the time we reach 12- years-old, but they are not fully mature until our mid-twenties – childhood, adolescence and young adulthood, sensing and physical development coupled in lock-step. Then it works for us seemingly without much thought, until we enter old age, when it starts to decay in different ways.

This journey of human sensing is what world modelmakers are aiming to develop, but it doesn’t stop there. Different animals sense the world around them in additional ways to humans, with stronger hearing, smelling and so on. Some have sensing capabilities we don’t, birds and sharks feel magnetic fields, bats are directed by sonar. We don’t even know these things are there.

This journey of human sensing is what world modelmakers are aiming to develop, but it doesn’t stop there.

Imagine that combinations of this sensibility could be hardwired from inception, into a mechanism that was also physically capable. A foundation world model that could predict what happens in a real-world geospatial environment, understanding cause and effect, space and time. Instead of taking 12 years to learn this through trial and error, it was pre-installed and fully formed, knowing the rules of physics. The Dojo scene from the film The Matrix, ‘I know Kung-Fu’ comes to mind. Autonomous vehicles are one example we see emerging in cities. Lots more will come.

Applying world models to autonomous, robotic systems will literally give them ‘a head start’. Less obviously, but more ubiquitously, and creating much greater scale, is ingesting world models into telecoms networks. All telecoms networks have sensing capabilities. They always have. They just didn’t have a frame of reference to know what they were sensing, it was just noise.

World models have the potential to turn networks’ experience of ubiquitous noise into signals. Signals can be turned into information, intelligence, and then could be used to communicate with autonomous systems for real world physical action. At world scale.

We live in a time of exponential change, surrounded by many uncertainties. Predicting when this will happen, five years, 10, 30? It’s hard to know. However, more and more people are seeing the roadmap emerging. I’m pleased to be joining fellow visionaries as we think about governance, who benefits, and what we should do next.

If it sounds interesting, please do join us at the UN’s 3rd Virtual World’s Day, 11-12 May. For more details, go to UN Virtual Worlds.