Augmented Reality Drone Overlays Feed Data to First Responders | Robotics

Recent hurricanes, wildfires and other natural disasters has led more public safety agencies to use commercial drones to aid them in search and rescue operations. Advances in hardware, software, and more trust from regulatory agencies are driving the growth of drones within the public safety sector.

The Center for the Study of the Drone at Bard College estimated that at least 910 state and local police, sheriff, fire, and emergency services agencies in the U.S. have acquired drones. The center said the number of public safety agencies with drones increased by 82% in the last year.

“All told, there are now more than twice as many agencies that own drones as there are agencies that own manned aircraft in the U.S.,” the center gave in its latest update around public safety drones.

Moving beyond an ‘eye in the sky’

While many agencies use drones solely to provide a video “eye in the sky” for their operations, new software is adding augmented reality to provide public safety officials with additional data and better situational awareness. A two-year-old startup called Edgybees has been helping public safety agencies with search and rescue efforts following disasters such as Hurricanes Florence, Michael, and Irma, as well as wildfires in California.

The company’s augmented reality (AR) software provides a real-time overlay to the video feed for a commercial drone, offering mapping data and the ability for officials to collaborate with additional first responders on the ground, instead of communicating solely through a two-way radio.

The company’s origins, however, were not in the public safety space. Founded in early 2017, the software was initially developed as an augmented reality video game for drone owners, where users could fly their physical drones through virtual hoops, or participate in a Death Star-like trench run.“The analogy I make is if you watched NFL games and you had John Madden scribbling over live video, saying, ‘Check out this guy over here,’ ” said Edgybees CEO Adam Kaplan. “Instead of broadcasting the world a football game, you’re broadcasting a scene that’s maybe a search and rescue, or a police scenario, or you’re giving information and awareness to a public safety official that understands where somebody might be trapped.”

“A number of commercial, public safety, and defense officials saw that we were the first company to do real-time AR on a drone, and they said, ‘One of the biggest issues in flying a drone is your situational awareness’,” Kaplan said. “So they came with us with requests to essentially be like Waze in the air, where you could overlay street maps and drop markers or other points of interest, as well as identifying people. That’s how the company took the technology and went in a different direction with it.”

Because the software can be used in multiple applications and markets, Kaplan said the company has received interest across the business and consumer landscape.

“We have a number of people who are taking our technology and using it for inspections, for construction, for first response, or defense,” Kaplan said. “We ended up building something that’s a bit like ARKit or ARCore, but not on a phone, but on a thing that’s flying around fast. I know it sounds like we’re in a number of verticals, but it’s really the same software, being consumed by a number of verticals.”

In addition to drones, Kaplan said the Edgybees software can work with video feeds from cameras on cars, helicopters, surveillance cameras, and even body cameras. He said they recently tested the software with a feed from an autonomous ground vehicle (AGV) as well.

In the automotive space, as cameras are added to new cars, Edgybees can take those feeds and provide contextualized information, not just for drivers, but for passengers as well. For example, a child in the back seat could use the video feed and then play a virtual Mario Kart-style game with the augmented overlay, Kaplan said.

Challenges and regulations

Like many in the unmanned aerial vehicle space, Edgybees works with customers that have regulatory issues or concerns. While the U.S. government continues to test drone integration programs with commercial drone operators, several regulations continue to affect the industry.

As an example, Kaplan said a customer in the broadcasting industry wants to use the software for sporting events, but that requires waivers around flying over spectators.

“It’s a lot easier to quickly turn on and turn off fly zones and no-fly zones,” Kaplan said. “But there are still significant barriers and issues in flying drones in certain areas.”

For example, in New York City, the dense population area and concerns around terrorism make operations there virtually impossible.

However, in the public safety space, where agencies can usually receive waivers more quickly than regular commercial operators, the biggest challenge is to keep up with the technology and add features to help with first response situations.

“The biggest challenge is that the drone industry is moving so quickly,” Kaplan said. “So we need to constantly develop and work together with customers to develop features alongside with customers. For example, we developed a tracker where a policeman or firefighter can put an app on their phone, with their own name on it, so their name and tracking information shows up on the drone video for the drone operations officials.”

Another feature lets drone pilots send text messages with the photo and markers from the video feed to officers on the ground, he added.

One of the reasons the company moved into the more commercial markets was a $5.5 million seed round of funding in February 2018, with participation from Motorola Solutions Venture Capital, Verizon Ventures, OurCrowd, 8VC, NfX, and Aspect Ventures.

With the additional funding, the company is growing its staff, now at about 25 people, along with support for the additional markets, Kaplan said. While finding new employees can be difficult in a tight labor market, he said the cool factor of the technology helps attract talent.

“The good news is that we’re dealing with pretty fun stuff like augmented reality, drones, and computer vision,” Kaplan said. “We’re on the cutting edge in terms of delivering technology that can really make a difference.”