The International Emergency Drone Organization (IEDO), an association for drone operations by rescue workers, in which first aiders around the world exchange their experiences, was founded in June 2018. In the IEDO annual report 2020 there is a lot of talk about camera support by flying drones, about infrared recordings, about image stabilization and flight search patterns. But it also says that the use of microphones on rescue drones is not possible today.
Macarena Varela, researcher at the Fraunhofer Institute for Communication, Information Processing and Ergonomics (FKIE), is currently proving the opposite with her experiments. Their flying drones, equipped with microphone arrays, block out wind and rotor noise and then accurately pick up signals from the environment. Varela’s development goal is drones that automatically detect calls for help, clapping or knocking in disaster areas, for example, and locate the position of the noise source.
A question of weight
In the first research approaches, Varela and her team relied on mobile systems, small unmanned tracked vehicles that can carry the superstructures made up of several analog microphones and an on-board computer. The researchers also experimented with microphone configurations for use on helicopters. Even with these approaches, it was clear that a bandpass filter must first filter out the wind and engine noise before the acoustic sensor data allow signals to be reliably recognized.
And it turned out that the weight is not insignificant: An array of eight condenser microphones put together in 2018 resulted in a structure of 50 cm × 90 cm and weighed together with the computer unit more than 10 kg.
But at the moment when colleagues at the institute examined the digital MEMS microphones (Micro electro-mechanical systems) in detail, Varela recognized the opportunity. The small digital microphones require little energy and no further signal conversion. In contrast to the analog microphones in their previous experimental setups, which were larger and heavier and required a stronger power supply and digitization of the recording, Varela was able to build much lighter-weight systems with MEMS technology. This makes it possible to detect sound signals with small drones in flight.
Despite its roaring rotors, the drone, which is equipped with microphones and a processing unit, is able to detect and locate calls and knocking noises.
(Image: Fraunhofer FKIE)
The researchers gradually expanded their microphone configurations to 16, 32 and now 64 MEMS microphones. For their arrangement, they referred to a development their institute had developed from radar location: the Crow’s Nest Array (CNA). The name refers to the lookout on the highest mast of old sailing ships, the so-called crow’s nest. From there, an observer has a clear view in all directions, and sensors in CNA configuration should also recognize signals from all directions equally well. The researchers had previously placed their microphones on a line. In this arrangement, however, the measurement results could not provide any information, for example, as to whether a sound event was sounding from the front or from the rear.
64 microphones the size of handball
In the CNA arrangement, the sensors are distributed spherically. The MEMS microphones used work omnidirectionally, which means that they pick up sound from all directions. Beamforming can be used to determine the direction of origin of the sound in the CNA arrangement by correlating the arrival times recorded with microsecond accuracy at the individual microphones and evaluating the time offset. This results in the bearing, which works with the same accuracy in every direction of space due to the spherical shape. For a drone with such a microphone array, this means that it can accurately determine the direction of a recognized noise in every flight position; it does not have to be exactly horizontal in the air.
Today, Varela’s team has the sophisticated CNA configuration of 64 MEMS microphones, including support structures made of fiber-reinforced plastic with integrated conductor tracks, produced by an industrial service provider who is based on a template drawing. This structure is only about 15 cm in diameter and, together with an FPGA card (Field Programming Gate Array) as a computing unit, weighs less than 1.5 kg. The researchers believe they can reduce the total weight to around one kilogram.
With this configuration, bearing is now possible with an angular accuracy of less than one degree. The researchers are currently patenting the method of sound localization using microphones in the crow’s nest array.
Background noise filtered out
The rotor noise, which tears on the nerves of the observer due to its pitch and volume, can be efficiently blocked out by a bandpass filter before the noise is detected. The on-board computer determines these background noises, but also wind noise, within a few milliseconds and then feeds the filter with the associated frequencies. However, it has been shown that the background noise under the drone is very variable. The drone generates different levels of noise, depending on whether it is hovering, i.e. hovering close to the ground, or flying higher, whether it is accelerating or getting caught in a gust of wind. The system therefore has to correct the filtering of the drone noise several times a second. It is particularly important for use in the event of a disaster to adapt the filter in real time.
The researchers are currently optimizing this system with GPS control. This means that they shorten the adjustment intervals when the drone changes its position and even more so when it changes its speed. The system also has to adjust the bandpass filter at short notice if the position changes. On the other hand, if the flight attitude is stable and the flight movement is constant, the interval is longer.
Recognize screams
The reliable detection of target noises is also crucial. Artificial intelligence should help to distinguish calls for help or impulse noises such as clapping and knocking from bird calls or a babbling brook. The researchers are currently building a database with examples of typical alarm signals, cries for help and other impulse noises, which are characterized by a rapidly increasing intensity. To do this, they use public databases such as the DCASE Challenge (Detection and Classification of Acoustic Scenes and Events), but they also record noises themselves; the members of the team can do their utmost when they call and shout.
The specially developed crow’s nest array hangs under the belly of the drone. 64 digital microphones at the ends of the brackets form a sphere – the array hears sound equally well from all directions.
(Image: Fraunhofer FKIE)
The aim is to use these sound examples to train a robust artificial intelligence that can distinguish human distress signals from other noises in the environment. In addition, this noise detection should also run in real time. In addition to different clapping and knocking noises, the database should contain at least 500 different examples of cries for help, the researchers estimate.
For the future, Varela plans to include a microphone for higher frequencies in its structure. This could help classify detected noises more precisely. With the previous system, it was already possible to recognize and locate emergency signals from people at distances of more than a hundred meters.
Search hit sent to the tablet
For rescue workers, the system is already able to display the drone data (latitude and longitude plus flight altitude) in front of a map background on a tablet computer. Currently, the drone and tablet are connected via WiFi. In addition, the drone can already transmit the bearing of sound sources. From this, specific locations can then be localized with several bearings.
As soon as an AI onboard classifies the sound sources as planned, the drone can concentrate on emergency signals. It should then only forward the bearing of selected search hits and can also transmit the result of the AI classification, i.e. whether it is, for example, a scream or a knock. It is even conceivable that the AI will learn even more precise results over time, such as whether a woman or a man has called for help.
In addition, the researchers are thinking of combining their acoustic system with other sensor systems such as cameras for visible light or infrared recordings. The automatic direction finding of a call for help can then also be used to align the cameras to the corresponding point. It is also conceivable that several drones support each other and locate a source of noise more quickly through their bearings.
The Fraunhofer are currently looking for partners for their acoustic search drones. After the tests in a real operational environment, the research service provider’s work ends, explains Dr. Kai Nürnberger. The team is now in contact with the THW and international networks such as the IEDO and Drone Responders. It is hoped to be able to start tests on their test sites and, if necessary, to gain access to a database with sound files of typical emergency signals from those seeking help.
Flying leak detectors
At the same time, the Fraunhofer are looking for partners from industry to implement their prototypes ready for series production. One advantage is that the MEMS microphones used are standard components that are used in millions of smartphones today and are mass-produced. The conceived microphone array could thus be produced at comparatively low costs. Nürnberger estimates that a ready-to-use prototype will be ready to go in a year. It would be possible to manufacture a finished system for series production within two years.
In addition to the possible uses in rescue operations, Nürnberger sees completely different opportunities to detect static or mobile noise on flying drones with acoustic sensors. These microphone arrays could detect vandalism at ATMs or locate approaching drones in the restricted area of an airport.
Another application that is only possible with an accurate bandpass filter is the automated search for gas leaks in industrial plants by drone. Inspection teams are often on the move in this area to track down a drop in compressed air or a gas leak. An automated drone with acoustic sensors designed for this purpose could fly off systems at regular intervals and detect leaks more quickly. In principle, noise detection is even easier to implement in closed rooms and halls, since no wind effects have to be taken into account there.
Ultimately, acoustically equipped search drones could detect any target noise and, geared towards this, fly their routine search laps in automatic mode and perform monitoring tasks. To do this, you only need to know the noises you are looking for and the typical ambient noises.
In c’t 23/21 you will learn about the hackers’ tricks. With the right tools you can crack forgotten passwords, find security gaps in your network and decrypt Word documents. We also analyze what makes WordPress so unique and help you get started and find suitable themes and plug-ins. You will find c’t 23/21 from October 22nd in Heise shop and at the well-stocked newspaper kiosk.
(agr)
