From Observation to Action: Advancing Autonomous Underwater Intervention

How research at the Australian Centre for Robotics is exploring the next generation of underwater robotic perception, manipulation and autonomy.

Underwater robotics has traditionally focused on observation, inspection and surveying. However, as marine operations become increasingly complex, research is expanding towards a new challenge: enabling robotic systems to physically interact with their surroundings with greater autonomy.

From biological sampling and environmental monitoring to subsea infrastructure maintenance and defence applications, future underwater missions may increasingly depend on robots capable of perceiving unfamiliar environments, interpreting spatial information and planning physical actions.

Achieving this requires advances in underwater perception, machine learning, robotic manipulation and control.

To support the development of these capabilities, Reach Robotics is investing in research undertaken by the University of Sydney's Australian Centre for Robotics, exploring the technical foundations of more autonomous underwater intervention.

Advancing Research in Underwater Robotic Autonomy

The University of Sydney's Australian Centre for Robotics has a long history of field robotics research, including autonomous underwater vehicles (AUVs), remotely operated vehicles (ROVs), underwater perception and robotic manipulation.

Led by Professor Stefan Williams, the marine robotics research group investigates applications ranging from high-resolution reef and habitat mapping to autonomous intervention, 3D reconstruction and learning-based robotic control.

Reach Robotics manipulators are used across research activities investigating how underwater robots can transition from observing their surroundings to interacting with them. Explore the Reach Alpha 5 robotic arms.

As Raghav Mishra, a researcher at the Australian Centre for Robotics, explains:

"Many of our current research questions actively use Reach Robotics manipulators or are aimed at advancing manipulation capabilities underwater."

Robotic manipulation introduces additional challenges in perception, planning, control and autonomy. For a robot to perform reliable intervention tasks, it must accurately perceive objects, estimate their position and orientation, plan safe movements and react to disturbances caused by currents and vehicle movement.

Understanding the Environment: Perception for Manipulation

One area of research focuses on perception for robotic manipulation — enabling a robotic arm to understand the three-dimensional environment in which it operates.

Unlike observation-based tasks, manipulation requires accurate spatial information about both the target object and the surrounding working volume. Even small errors in depth estimation can affect a robot's ability to position its manipulator and execute a grasp.

Yiran Wang, PhD researcher at the University of Sydney's Marine Group, describes the importance of this distinction:

"Manipulation has not influenced my research direction. It is the research direction."

Underwater perception presents considerable technical challenges. Lighting is weak, uneven and attenuates differently to in air. The water scatters light back into the lens, leaving images blurred and low in contrast. Man-made structures and the marine growth covering them often create broad areas of visually repetitive texture. Put together, there are very few usable matching cues in the image.

That also constrains the choice of sensor. Active approaches each carry a cost here: structured light and time-of-flight have their patterns broken up by scattering, sonar lacks the resolution for close-range manipulation, and this equipment tends to be bulky and power-hungry. The university works with a passive stereo system, two cameras only, which keeps size and power low and gives true scale directly. The cost is that every difficulty is pushed onto the algorithm.

Then there is data. There is no data flywheel underwater: real data is expensive to collect and scarce, and data with accurate 3D ground truth scarcer still, while synthetic data can be generated without limit but has to cross the sim-to-real gap. As a result, the pressure lands on generalisation. Trained on limited data with a narrow distribution, a model overfits easily to one kind of water and one kind of lighting and fails in the next scene, and trading accuracy away for generalisation leaves it short of what manipulation requires. Generalisation and accuracy usually pull against each other and underwater manipulation needs both.

Yiran's research explores these challenges through LinStereo, a stereo depth model designed for close-range underwater imagery.

Trained entirely on synthetic above-water data, LinStereo transfers zero-shot to real underwater environments without having encountered underwater imagery during training.

On public underwater datasets, the model reduces depth error by 28% on TartanAir-UW and 26% on SQUID. Laboratory experiments have also demonstrated its ability to recover fine structures, including thin ropes, at distances below two metres.

Beyond depth estimation, the research explores SLAM (Simultaneous Localisation and Mapping) and how observations from multiple viewpoints can be fused into a unified three-dimensional representation using a truncated signed distance field (TSDF).

This spatial representation can then support the prediction of six-degree-of-freedom (6-DoF) grasp poses, connecting environmental perception with potential robotic actions.

Together, these research directions address a fundamental challenge: enabling a robot to construct a sufficiently accurate understanding of an unfamiliar environment to support physical intervention.

From Perception to Autonomous Action

While accurate environmental perception is essential, it represents only one component of autonomous underwater intervention.

Robots must also determine how to approach objects, execute appropriate movements and adapt when environmental conditions change.

Researchers are investigating how machine learning techniques, including imitation learning and vision foundation models, could support these capabilities.

A key area of interest is sim-to-real transfer — the ability to develop and train models using simulated data and apply them effectively in real-world environments.

This is particularly relevant underwater, where collecting sufficiently diverse, accurately labelled training data can be expensive and technically difficult.

Advances in these areas could gradually reduce the need to develop task-specific models for every new environment, helping robotic systems adapt to unfamiliar objects and operating conditions.

As Raghav Mishra explains:

"This also enables downstream capabilities in physical interaction and intervention rather than just perception, making manipulation more important than ever."

Such developments could support collaboration with marine scientists and industry partners. Beyond inspection and surveying, we can now explore applications such as biological sampling, environmental monitoring, infrastructure maintenance, and autonomous marine asset intervention. These applications raise new research questions around dexterous manipulation in unstructured environments, human-robot collaboration, shared autonomy, and how foundation models and learning-based approaches can be used to enable more capable underwater robots.

Exploring New Opportunities in Deeper Waters

As underwater robotic systems become capable of operating at greater depths, new opportunities emerge for scientific exploration, environmental research and subsea intervention.

Operations in a mesopelagic zone, for example, can involve unfamiliar environments, limited opportunities for direct observation and increasingly demanding deployment requirements.

Depending on the vehicle and communications architecture, underwater operations may also face bandwidth constraints, limited operator feedback and restrictions on real-time supervision.

These considerations create important research questions around how robotic systems perceive, plan and act when operating with incomplete information.

Yiran highlights another challenge associated with deeper environments:

"Deep-sea targets have no CAD model and many have never been seen by anyone, so a grasp has to be inferred from geometry reconstructed on the spot."

For researchers, this raises questions about how robotic systems can reconstruct unfamiliar objects, identify suitable grasp points and safely interact with fragile or deformable materials, including biological samples and loose sediment.

Potential applications extend from deep-water biological sampling and habitat monitoring to inspection and physical intervention around subsea infrastructure.

Rather than treating autonomy as an absolute requirement at a particular depth, ongoing research is exploring where greater onboard intelligence, improved perception and autonomous decision-making could expand operational capabilities.

Shaping the Future of Underwater Intervention

The next generation of underwater robotics will depend not only on how effectively robots observe their surroundings, but also on how reliably they translate environmental information into physical action.

Advances in stereo vision, 3D reconstruction, machine learning and robotic manipulation are helping researchers investigate the technical foundations of more capable autonomous intervention.

Through its continued investment in research at the University of Sydney, Reach Robotics is supporting the exploration of these capabilities and their potential applications across increasingly complex underwater environments.

As the boundaries of underwater exploration and intervention continue to evolve, connecting perception, autonomy and manipulation will be an important step towards enabling robotic systems to undertake a broader range of tasks beneath the surface.