U.S. Secretary of Transportation Elaine Chao has launched an initiative to safely test and validate advanced drone operations in partnership with state and local governments in select jurisdictions.
Prospective local government participants are asked to partner with the private sector to develop pilot proposals. After evaluating all of the applications, USDOT will invite a minimum of five partnerships.
The department also will publish a Federal Register Notice with more details about how applications will be evaluated and how the program will work.
More about the program is available on the DOT website.
The program will help tackle the most significant challenges in integrating drones into the national airspace while reducing risks to public safety and security, USDOT said. The program is designed to provide regulatory certainty and stability to local governments and communities, UAS owners and operators who are accepted into the program.
In less than a decade, the potential economic benefit of integrated unmanned aerial systems into the nation’s airspace is estimated to equal up to $82 billion and create up to 100,000 jobs, according to an economic report by the Association for Unmanned Vehicle Systems International (AUVSI).
The program will help the USDOT and Federal Aviation Administration (FAA) develop a regulatory framework to:
allow more complex low-altitude operations;
identify ways to balance local and national interests;
improve communications with local, state and tribal jurisdictions;
address security and privacy risks; and
accelerate the approval of operations that currently require special authorizations.
“This program supports the president’s commitment to foster technological innovation that will be a catalyst for ideas that have the potential to change our day-to-day lives,” Chao said. “Drones are proving to be especially valuable in emergency situations, including assessing damage from natural disasters such as the recent hurricanes and the wildfires in California.”
The pilot program will evaluate a variety of operational concepts, including night operations, flights over people, flights beyond the pilot’s line of sight, package delivery, detect-and-avoid technologies, counter-UAS security operations, and the reliability and security of data links between pilot and aircraft.
Industries that could see immediate opportunities from the program include commerce, photography, emergency management, precision agriculture and infrastructure inspections and monitoring.
“Stakeholders will have the opportunity through this program to demonstrate how their innovative technological and operational solutions can address complex unmanned aircraft integration challenges,” said FAA Administrator Michael Huerta. “At the same time, the program recognizes the importance of community participation in meaningful discussions about balancing local and national interests related to integrating unmanned aircraft.”
Organized by Professor Zak Kassas from the University of California, Riverside, the workshop will feature presentations and panels by experts and leaders from government (National Science Foundation, Office of Naval Research, Air Force Research Laboratory, Department of Transportation), industry (Google, Daimler, and Ford) and academia (The Ohio State University, UC San Diego, University of Southern California).
The workshop will discuss opportunities and challenges (technical, commercial, ethical, and legal) associated with developing fully autonomous systems that are cognizant and trustworthy for safety-critical applications. Examples include unmanned aerial vehicles (UAVs), self-driving cars and unmanned underwater and surface vehicles.
Kassas, director of the Autonomous Systems Perception, Intelligence, & Navigation Laboratory (ASPIN), leads a team of researchers developing reliable and accurate navigation that exploits existing signals of opportunity, rather than GPS, to meet the stringent requirements of fully-autonomous systems, such as UAVs and self-driving cars.
Septentrio debuted the AsteRx-m2a and AsteRx-m2a UAS GNSS OEM engines at Commercial UAV 2017, held Oct. 24-26 in Las Vegas.
The two new OEM boards provide precise and reliable multi-frequency, all-in-view real-time kinematic (RTK) positioning and heading — along with interference technology — with low power consumption, the company said.
Both boards are smaller than a credit card and feature Septentrio’s AIM+ interference mitigation and monitoring system. AIM+ can suppress a wide variety of interferers, from simple continuous narrowband signals to the most complex wideband and pulsed jammers.
The AsteRx-m2a board by Septentrio. Photo: Septentrio
Increasing levels of radio-frequency pollution, coupled with the intrinsic danger of self-interference in compact systems such as UAS, makes interference mitigation a vital element in any UAS system that uses GNSS positioning.
Both boards are designed to bring high-precision positioning and attitude to any space-constrained application. According to the company, both receivers are designed to serve as core components in any multi-sensor application.
The AsteRx-m2a UAS is aimed specifically at unmanned applications, bringing plug-and-play compatibility for autopilot systems such as ArduPilot and Pixhawk. Event markers accurately synchronize camera shutter events with GNSS time. The board can be powered directly from the vehicle power bus via its wide-range input.
The AsteRx-m2 UAS board by Septentrio. Photo: Septentrio
The AsteRx-m2a UAS works seamlessly with GeoTagZ software, providing offline re-processed RTK accuracy without the need for either ground control points or a real-time datalink.
“We’ve taken the hugely successful AsteRx-m2 and added a second antenna input for high-precision GNSS heading,” said Gustavo Lopez, OEM product manager at Septentrio. “No need to manoeuvre around in a figure of ‘8’ trying to initialise INS heading or find space or additional power for a separate INS module now. All you need is a second antenna and you’re good to go.”
Septentrio is located at booth 206 of Commercial UAV Expo 2017.
It’s been a couple of months since we ran an update on unmanned aircraft, so there are lots of news items to dust off and maybe look at more closely.
I suppose we’ve all seen those futuristic movies with masses of orderly air traffic traveling rapidly down invisible roads hundreds of feet above cities — maybe the Jetsons first got us thinking about this vision of tomorrow? Well, unmanned flying taxi demos in Dubai certainly caught my attention. Could this be the launch of the first phase of “airborne street traffic”?
Demo UAVs in Dubai, China
The two-seater UAV built by Volocopter demonstrated in Dubai has 18 rotors, and during the five-minute demo for the media, Crown Prince Sheikh Hamdan bin Mohammed was flown at around 200 meters over sand, rather than over a populated city. There surely could be a number of safety elements yet to be implemented before we see this become operational — but you have to start somewhere.
The Volocopter demo was preceded at the beginning of this year by the appearance of a single-seat Chinese demo vehicle. This smaller eight-rotor drone by EHang took a shot at being a future “over-city” cab.
Urbain Air Project
In the meantime, Airbus and HAX, a start-up investor, are seeking innovators to participate in a four-month program to advance developments in urban-air mobility — innovations which could speed-up development of “flying cars.”
The project is looking for technologies already being developed in:
Urban air transport vehicle technology
UAV sense and avoid technology
Airport runway and landing detection systems
Emergency safety systems for airborne vehicles
Required infrastructure for airborne transport vehicles
Autonomous airborne vehicle technology
Aerial maneuver decision making and support systems
Air traffic management systems
Aerial collision detection and avoidance systems
Battery packaging and management systems for airborne vehicles
Several startups could be funded with at least $100,000 each, and will be asked to spend four months in Shenzhen, China, turning their concepts into prototypes with support from HAX and Airbus engineers.
Safety Standards?
All interesting stuff, but at some stage someone has to take a serious look at the safety standards needed to protect prospective passengers. The existing designs appear to have some flight control redundancy, and there are hints of a possible loss of data-link reversionary mode, but there might be more significant work to be done before any regulatory agency such as the Federal Aviation Administration (FAA) were to validate system reliability. But good luck to these innovators and other companies who are working towards implementing this fascinating concept.
At the other end of the drone spectrum, Renishaw Canada recently showed off a drone made of titanium and produced using 3D printing.
The Firefly is a 3D-printed titanium rocket-powered drone that can fly at nearly supersonic speeds, with onboard telemetry and a spring-released wing. The Mach 0.8 drone has been produced by the Renishaw additive manufacturing group for an unnamed North American aerospace company. The drone can apparently house a number of miniaturized sensors for data collection.
Possible applications of this unique high-speed, short-duration drone could include data collection flying into storms and hurricanes, or perhaps for longer distance surveying when launched from a future Mars rover.
Boeing Acquires Aurora Flight
And on the business front, the recent news is that Boeing is in the process of acquiring Aurora Flight Sciences Corp. Adding Aurora as an independent operation alongside Insitu will probably lead to migration of technology between the two Boeing UAS units, which is presumably why Aurora is being acquired.
Aurora has focused on electric propulsion systems and automation and autonomy for robotic operations and UAVs. Aurora has also collaborated with Boeing in the past on rapid prototyping for drones, and structural assemblies for military and commercial applications.
As a unit of the Boeing Company, Aurora technologies for long-endurance aircraft, robotic co-pilots, and autonomous electric multi-rotor UAVs will have a better opportunity to make it to product level, and wider applications should be possible for these unique capabilities.
Based in Manassas, Virginia, with facilities and offices in five other states around the United States — including R&D facilities right next to Massachusetts Institute of Technology in Cambridge — Aurora employees more than 550 people. They also have an office in Luzern, Switzerland.
FAA Regulations Revisited
Finally, according to AUVSI, in the year since the FAA released the Part 107 regulations for the operation of small UAS (sUAS), users have requested more than 1,000 waivers to work outside the parameters of these regulations. The Part 107 regulations permit users to request such waivers, provided operations can be shown to be safe. The majority of these waiver requests were to operate at night — whereas the regulations only permit operation within Visual Line of Sight (VLOS) in daylight.
AUVSI argues that certain commercial operations have only been possible through the use of these waivers, and therefore the regulations should be revised to enable normal operations without the need to grant individual waivers on a case-by-case basis. The FAA’s position may be that until such operations can be proven to be safe over time, the agency wants to know who’s exceeding which parameters, and under what conditions — hence the need for individual written applications, so that analysis of safety aspects is possible. Then subsequent monitoring will show that levels of operation may be safely exceeded on a regular basis.
This is how aviation agencies have always managed aviation safety. A UAS operator might demonstrate operational capabilities, show an acceptable safety level, and thereby prove that pushing the envelope is okay. Sometimes it can take time, but with good visibility on both sides, it’s possible that progress could be made reasonably quickly.
The Transportation Safety Board of Canada (TSB) is conducting an investigation into the collision between a drone and a passenger aircraft that took place on approach to the Jean Lesage International Airport in Québec City on Oct. 12.
On that day, a Beech King Air A100 operated by Skyjet M. G. was on an instrument flight rules flight from the Rouyn-Noranda (Quebec) airport to the Jean Lesage International Airport in Québec City with two crew members and six passengers on board.
The aircraft was approaching runway 24 and had just passed the final approach fix when the crew noticed an unmanned aerial vehicle (UAV) off the left wing. The aircraft struck the UAV at an altitude of 1500 feet and the crew declared an emergency.
Aircraft rescue and firefighting services were deployed and the aircraft safely landed on runway 24. The aircraft inspection revealed a few scratches and some paint transfer on the top surface of the left wing and scrape marks on the de-icing boot.
The aircraft was then returned to service. No one was injured.
Javad GNSS’ Javad Ashjaee offers a rundown on the company’s Triumph-F1 unmanned aerial vehicle at Intergeo 2017, which took place Sept. 26-28 in Berlin, Germany. According to the company, the Triumph-F1 is a field-tested high-precision geodetic GNSS receiver that includes four battery compartments, four angled documentation cameras and more.
Commercial drone-maker Delair has introduced a professional unmanned aerial vehicle (UAV) for survey-grade photogrammetric mapping.
The UX11 is a small fixed-wing UAV that combines a powerful integrated onboard system, industry-grade sensors, limitless communication range and PPK centimeter-level positioning. It carries enough onboard computing power to access and process the pictures, then sends them to the operator in real-time.
According to the company, it will run automated quality checks on the images (such as blur detection or overlap checks) to help ensure the operator is acquiring quality data.
The UX11’s redundant communications system includes a proprietary line of sight radio and 3G/4G connectivity between the ground control station and the UAV using a worldwide machine-to-machine pre-paid plan.
Building on Delair’s experience with beyond visual line of sight (BVLOS) operations since 2012, the UX11 is ready for BVLOS flights with unlimited range and adds a new level of safety with this communication link.
The UX11 is lightweight, ultra-stable, simple to hand-launch at takeoff and it lands precisely where planned using distance measuring technology. New user-friendly Android mission planning software boasts innovative features such as support for in-flight camera feedback and live data review, the company said.
Made to help professionals in GIS, survey, and construction optimize area coverage per flight, the UX11 flies for 59 minutes with the best coverage and resolution specifications in its class for flights at 122 m (400 ft) altitude above ground level. The UX11 will be available for purchase via DELAIR’s global network of distributors by January, 2018.
The UX11 is a product offer for data acquisition which can be complemented by data processing and analytics software programs to address a range of commercial applications. Geospatial users can create 2D and 3D models and then generate elevation profiles, contour lines, slope qualifications and volumetric estimates with high accuracy and resolution using post-processed kinematic data and ground-control points.
Professional drone company Wingtra is partnering with photogrammetry company Pix4D. Pix4D’s software suite is now available to WingtraOne users, both directly and via Wingtra’s distributors.
WingtraOne, Wingtra’s main product, is a vertical take-off and landing (VTOL) UAV that enables data collection for a variety of industries. The partnership with Pix4D aims to augment its status with an end-to-end solution including 2D map and 3D model construction from aerial data.
The WingtraOne drone bridges the gap between traditional multi-rotors and fixed-wing drones, the company said. It takes off and lands vertically like conventional multirotors, but once in flight, the drone tilts forward to fly like a fixed-wing aircraft.
Being able to carry heavy payload such as the Sony RX1RII, the drone offers high mapping accuracy, while covering an area of 980 acres (400 Ha) at 3 cm/px (1.2 in/px) GSD or the equivalent of 570 football fields.
The WingtraOne is available in use in Europe, China, the United States and Australia for applications ranging from surveying and precision agriculture to glacier monitoring.
Wingtra (booth 109) and Pix4D (booth 415) are exhibiting at Commercial UAV Expo Americas, which takes place Oct. 24-26 in Las Vegas.
Map made by Pix4D pictures taken by WingtraOne with RX1RII camera. (image: Wingtra)
Turning Information into Insight. Wingtra’s diverse user base is complemented by Pix4D, whose product range is aimed at the surveying and agriculture industry, among others.
Pix4D has allows professionals to generate high-quality point clouds, orthomosaics, surface and terrain models from aerial imagery. Some of its popular offerings include Pix4Dmapper for precisely georeferenced 2D maps and 3D models, and Pix4Dag for accurate reflectance and index maps (NDVI, NDRE).
With WingtraOne’s autonomous aerial data collection and Pix4D’s advanced data-analysis capabilities offered as a single bundle, professional users can now expect a plug-and-play solution. “We are keen on collaborating strongly in our upcoming events. Actually we are meeting very soon at UAV Expo in Las Vegas,” Bailey said.
“The bond between the companies was established some time ago, since realizing the potential of pairing high-resolution aerial images with cutting-edge photogrammetry modeling software,” said Caroline Bailey, Pix4D regional sales manager for Europe. “We are very happy to announce the decision to become official partners.”
Leopold Flechsenberger, sales manager at Wingtra, added, “We have always aimed at providing the best survey-grade aerial imagery to our users, so Pix4D was an obvious choice from the start. From now on, Wingtra is offering a reduced price on WingtraOne drones, when bundled with Pix4Dmapper.”
AeroScope addresses safety, security and privacy concerns while protecting drone pilots
DJI has unveiled AeroScope, its new solution to identify and monitor airborne drones with existing technology that can address safety, security and privacy concerns.
AeroScope uses the existing communications link between a drone and its remote controller to broadcast identification information such as a registration or serial number, as well as basic telemetry, including location, altitude, speed and direction.
Police, security agencies, aviation authorities and other authorized parties can use an AeroScope receiver to monitor, analyze and act on that information. AeroScope has been installed at two international airports since April, and is continuing to test and evaluate its performance in other operational environments.
“As drones have become an everyday tool for professional and personal use, authorities want to be sure they can identify who is flying near sensitive locations or in ways that raise serious concerns,” said Brendan Schulman, DJI’s vice president for policy and legal affairs. “DJI AeroScope addresses that need for accountability with technology that is simple, reliable and affordable — and is available for deployment now.”
DJI demonstrated the system Oct. 12 in Brussels, Belgium, showing how an AeroScope receiver can immediately sense a drone as it powers on, then plot its location on a map while displaying a registration number. That number functions as the equivalent of a drone license plate, and authorities can use it to determine the registered owner of a drone that raises concerns.
In March 2017, in response to growing calls by governments worldwide for remote identification solutions, DJI released a white paper describing the benefits of such an approach to electronic identification for drones.
AeroScope works with all current models of DJI drones, which analysts estimate comprise more than two-thirds of the global civilian drone market. Since AeroScope transmits on a DJI drone’s existing communications link, it does not require new on-board equipment or modifications, or require extra steps or costs to be incurred by drone operators. Other drone manufacturers can configure their existing and future drones to transmit identification information in the same way.
Because AeroScope relies on drones directly broadcasting their information to local receivers, not on transmitting data to an internet-based service, it ensures most drone flights will not be automatically recorded in government databases, protecting the privacy interests of people and businesses that use drones. This approach also avoids substantial costs and complexities that would be involved in creating such databases and connecting drones to network systems.
This system is consistent with DJI’s problem-solving approach to drone regulation, which aims to strike a reasonable balance between authorities’ need to identify drones that raise concerns and drone pilots’ right to fly without pervasive surveillance.
DJI has led the industry with safety and security advances such as geofencing and sense-and-avoid technology, and believes the rapid pace of innovation provides the best means to address new policy concerns.
Drone identification settings will be included in DJI’s initial drone software to allow customers to choose the content of their own drone’s identification broadcast to match local expectations both before and after identification regulations are implemented in different jurisdictions.
To protect customers’ privacy, the AeroScope system will not automatically transmit any personally identifiable information until regulations or policies in the pilot’s jurisdiction require it.
“The rapid adoption of drones has created new concerns about safety, security and privacy, but those must be balanced against the incredible benefits that drones have already brought to society,” said Schulman. “Electronic drone identification, thoughtfully implemented, can help solve policy challenges, head off restrictive regulations, and provide accountability without being expensive or intrusive for drone pilots. DJI is proud to develop solutions that can help distribute drone benefits widely while also helping authorities keep the skies safe.”
Drones are quickly becoming a staple of the maritime industry. In January, the European Maritime Safety Agency (EMSA) issued the largest ever civilian maritime drone contact, valued at €67 million.
Under the contract, drones will be used to assist with border control, search-and-rescue operations and monitoring of pollution, as well as the detection of illegal fishing and drug and people trafficking.
External Vessel Inspections. Big names in the maritime industry such as DNV-GL, Lloyds Register and Maersk have all shown strategic intent to revolutionize their operations by embracing drone technology, and many maritime operators are now following suit.
All ship owners know that traditional methods of external vessel inspection can be a costly affair. Now that high-definition, camera-equipped drones are widely available and affordable, it is becoming more common to use them for external vessel inspections to assess structural conditions. Identifying substantial corrosion, significant deformation, fractures, damage or other structural deterioration can be done quickly, easily and cost-effectively using drones.
Tank Inspections. The visual inspection of cargo tanks was traditionally performed by workers suspended on ropes to inspect the tank structure. The sheer size of modern-day vessels means that access methods including staging, rafting and climbing are often used by surveyors to access tanks.
In contrast, drone surveys require no human access to the tank and, since no access equipment is required, there are no setup costs, and inspections can be completed within a quicker timeframe.
Martek Marine’s V-200 UAS. (Photo: Martek Marine)
Bathymetric Surveys. Accurate and reliable information on the features of water bodies and their shorelines is vital to navigational safety. Bathymetric surveys gather the information, which is then published for use on nautical charts. Rather than using a fixed-wing airplane or helicopter, bathymetric sensors developed for drones allow this type of survey to be carried out flexibly and at a fraction of the cost.
To operate effectively in the harsh maritime environment, the technology has been developed to withstand storm force wind and heavy rain, snow and salt spray.
As technology advances, so does the flight time available on drones, meaning more area can be covered in a quicker timeframe.
Floating Flare-Tip Inspections. Drone surveys typically exist to provide close visual and thermal inspections of high, live or difficult to access structures offshore, and there’s nothing more challenging to access than a flare tip, 70 meters above water, on a floating production facility.
Drone survey inspections for flare tips remove the need for a shutdown to inspect the flare and offer reduced costs compared to aerial surveys carried out by helicopter or plane.
Offshore Wind Energy. The wind energy sector is growing fast. Storm force winds, erosion, lightning strikes and even build-up of insects can have an impact on turbines, and blades need to be inspected for deterioration. Inspectors have traditionally had to scale the turbines with the help of ropes and cables.
The maritime surveying company Martek Marine uses a drone fleet designed for turbine-blade inspections onshore or offshore. Qualified and trained pilots quickly and accurately identify and assess faults.
Traditional surveying requires turbines to be offline for two hours up to a day, but Martek’s inspection process reduces this time to 45 minutes.
Following the inspection, the client can access the data through Martek’s secure, cloud-based asset management portal where they can download a detailed PDF report and access raw survey data.
Fully Autonomous Drones? Fully autonomous drones could be the next big thing for maritime surveying. The drones can be pre-loaded with a 3D model of the ship. This allows the drone to autonomously work its way around the vessel, stopping at points of interest to obtain detailed video or image data.
Advancing this further, a drone could be designed to create its own 3D map of the vessel before carrying out the survey independently.
This article is excerpted from a blog by Martek Marine, a UK-based maritime surveying company. Read the full blog, with more details and examples.
When a platform’s mission requires maneuvering among different environments, transitions between these environments may mean that a single method cannot solve the full positioning, navigation and mapping problem.
This article describes an integrated navigation and mapping design using a GPS receiver, an inertial measurement unit, a monocular digital camera and three short-to-medium range laser scanners.
By Evan T. Dill and Steven D. Young, NASA Langley Research Center, and Maarten Uijt de Haag, Ohio University
An unmanned aircraft system (UAS) traffic management system (UTM) is an ecosystem for coordinating UAS operations in uncontrolled airspace, particularly operations under 400 feet altitude involving small- to mid-sized vehicles. In this domain, information services regarding the state of the airspace will be provided to UAS operators.
In addition, UTM would coordinate and authorize access to airspace for particular time periods based on requests from the operators. The Federal Aviation Administration would maintain regulatory and operational authority, and may for example, issue changes to constraints or airspace configurations to operators via this information service. However, there is no direct control from air traffic control personnel (such as “climb and maintain 300 feet” or “turn left, heading 150”).
As with visual flight rules operations of manned aircraft in uncontrolled airspace, under UTM the onus is on the vehicle operator to assure the flight system provides adequate performance with regard to communication, navigation and surveillance during flight. The vehicle/operator is responsible for avoiding other aircraft, terrain, obstacles and incompatible weather. UTM information services do not yet include, for example, information from an alternative positioning, navigation and timing system that may be needed for operations conducted in GPS-degraded environments (such as near buildings or other structures). This is the challenge being addressed by the integrated navigation concept described in this article. Other concepts are also being considered and developed for alternate, and unique, UAS missions and flight environments.
The method presented here employs a monocular camera as part of a multi-sensor solution that continuously operates throughout and between outdoor and structured indoor environments. For this work, an indoor environment is considered “structured” if its walls are vertical and remain approximately parallel, while the floor is either roughly flat or slanted.
In this type of environment, GPS is typically only sparsely available or not available at all. Hence, in our proposed navigation architecture, additional information from a camera and multiple laser range scanners (not the focus of this article) are used to increase the system’s positioning, navigation and mapping availability and accuracy in a GPS-challenged indoor environment. Figure 1 shows the target operational scenario, and Figure 2 the equipped multi-copter used in this research.
Figure 1. Operational scenario: open-sky environment, transition to indoor and indoor environment.Figure 2. Hexacopter sensors and sensor locations.
Figure 3 shows a block diagram for the methodology implemented in this research, with the elements related to monocular camera methods highlighted. When assessing the capabilities of each of the sensors used in the work, only the inertial sensor produces data that is solely dependent on the motion of the platform and local gravity and is more or less unaffected by its surroundings. Therefore, the inertial is chosen to be the primary sensor for this method.
The mechanization integrates the measurements from GPS, the laser scanners and the monocular camera through a complementary Kalman Filter (CKF) that estimates the errors in the inertial measurements and feeds them back to the inertial strapdown calculations. For this inertial error estimation method to function properly, pre-processing methods must be implemented that relate the sensors’ observables to the inertial measurements.
Here we describe the processing techniques necessary to relate measurements from a monocular camera to measurements from the inertial measurement unit (IMU). Then we show how these techniques are used in the broader GPS/optical/inertial mechanization and present testing results.
Figure 3. Monocular camera components of a broader mechanization.
2D Monocular Camera Methods
To process data from the camera, we first perform feature detection and tracking of both point features and line features. Specifically, elements from Lowe’s Scale Invariant Feature Transforms (SIFT) are used to track point features, which are in turn used to obtain estimates of the camera’s rotational and un-scaled translational motion using structure from motion (SFM) based methods. To resolve the ambiguous scale factor, a novel scale estimation technique is employed that uses data from the platform’s horizontally scanning laser. This technique as well as algorithms that produce a 3D visual odometry solution are presented below.
SIFT Point Feature Extraction. To aid in determining camera motion, SIFT has been used as a way of identifying local features that are invariant to translation, rotation, and image scaling. This technique yields 2D point features that are unique to their surroundings and readily identified and associated across a set of sequential camera images. Each key location and its surroundings are analyzed, resulting in a descriptive 128-element feature vector, known as a SIFT key. Example results of the SIFT key identification process are shown in Figure 4.
Figure 4. SIFT feature identification.
Based on the results of the SIFT feature extraction process from two image frames, a feature association function is performed using the feature vectors. For this work, a two-step procedure is implemented.
First, SIFT keys are associated using a matching procedure. Example results of this process are shown in Figure 5, where it can be observed that incorrectly associated features may result from this process. To remove these artifacts, inertial measurements are utilized to ensure the correctness of the associations.
Figure 5. SIFT matching results between consecutive image frames.
Using a triangulation method, prospective associations are used to crudely estimate each feature’s 3D position with respect to the previous frame. While this triangulation method yields 3D data, it is of poor quality, and is therefore only used to obtain rough approximations that are sufficient for association purposes, but insufficient for navigation purposes.
Once transformed to a 3D reference frame, the projected distances of each feature are compared with one another, and prospective associations that produce significantly different depths than surrounding points are eliminated. Example results of this filtering process can be seen in Figure 6.
Figure 6. Point feature association after inertial based miss-association rejection.
In future implementations, the ORB feature will be evaluated, as its performance is expected to be more than two orders of magnitude faster than SIFT.
Wavelet Line Feature Extraction. To implement the scale factor estimation technique described in a later section, it is necessary to first extract and track vertical line features. To accomplish this, a method using wavelet transforms (WTs) was developed. When applied to a 2D image, WTs can be viewed as filters operating in the x and y directions of an image. By applying either a high- or low-pass filter to both of an image’s channels (that is, x and y directions), four sub-images are formed to represent an image approximation. For this work, a level-one bi-orthogonal 1.3 wavelet was used to decompose each image. An example of the four sub-images produced by this wavelet is shown in Figure 7 along with the original image.
Figure 7. Example results of wavelet decomposition.
Through further processing of the vertical decomposition results, strong line features are identified by first inspecting the illuminated elements along the vertical channels of the decomposed image and identifying clusters of adjacent pixels. Next, a 2D line fit is applied to the groups to estimate residual noise. Pixel collections with low residuals (<3 here) are considered valid line features. Example results of this process are shown in Figure 8.
Figure 8. Example vertical line extraction results.
For association purposes, lines cannot be compared over a sequence of image frames solely based on location as similar line features may not necessarily possess the same endpoint, and, therefore, can be of varying lengths. However, corresponding lines will possess many common points and similar orientations if they are projected into the same frame. Using the inertial reference frame, each line’s orientation, , can be transformed across image frames as given by:
In this manner, lines between frames that contain multiple similar points and have comparable orientations are considered associated.
For a discussion of the projective visual odometry and epipolar geometry methodology as well as the resolution of true metric scale used in this work, download the supplemental PDF.
Metric Scale. As the unscaled translation estimate calculated through the aforementioned visual odometry method is a unit vector, it only indicates the most likely direction of motion of the camera. To obtain the sensor’s actual translational motion, an estimate of the scale factor, m, is required to determine the absolute translation ∆r. This can be accomplished through techniques using a priori knowledge of the operational environment or measurements from other sensors. In this research effort, a new method is employed that makes use of data provided by a horizontally scanning laser.
The proposed method estimates the scale in an image by identifying points in the environment that are simultaneously observed by the camera and the forward-looking laser range scanner.
To enable this estimation method we must identify the correspondences between the pixels in the camera images (each defined by a direction unit vector corresponding to the row x and column y) and the laser scanner measurements (each defined by direction unit vector). A calibration procedure establishes these correspondences. Given the laser range measurements, 2D features located on the scan/pixel intersections can be scaled up to 3D points.
Unfortunately, extracted 2D point features are rarely illuminated by a laser scan in two consecutive frames. This can be resolved by considering the intersection of a laser scan with 2D line features rather than point features. As the laser intersects the camera frame at the same location regardless of platform motion, and the platform does not make excessive roll and pitch maneuvers, vertical line features in the image frame are preferred as they will be relatively orthogonal to the laser scan plane.
Using the previously described vertical line extraction procedure, Figure 9 shows an example image frame overlaid with the points in the image frame illuminated by the laser (indicated by a blue line) and the extracted vertical line features (indicated as green lines). Multiple intersections of 2D vertical lines with laser scan data are calculated (indicated as red points). Inversely, Figure 10 depicts the location of all laser scan points in green, all laser points observable with the camera field-of-view (FoV) in blue, and intersection points in red.
Figure 9. Image frame overlaid with points; Laser (blue), vertical line features (green), multiple intersections (red).Figure 10. 2D vertical line and laser intersections in laser scan data.
For scale factor calculation purposes, it is necessary to track the motion of these 3D laser/vision intersection points, across sequences of camera image frames. As each intersection point uniquely belongs to a line feature in the 2D image frame, it can be stated that if two lines are associated, their corresponding intersection points are also associated. Using the rotation computed from the visual odometry process, the line association method described by (1) is implemented, and provides associations between laser/vision intersection points across frames.
To calculate the desired scale factor based on these associated laser/vision points, geometric relationships are established: unit vectors from the camera center to points located on a 2D line. From these, the line’s normal vector can be derived.
Monocular Camera Results
To assess the performance of the visual odometry processes, multiple experiments were conducted. The results of one such test are discussed here. During each test, the visual odometry results for rotation, shown in blue, were easily evaluated through comparison with the platform’s inertially-measured rotation, displayed in red.
The rotational results for each sensor were decomposed into the Euler angles: pitch, roll and yaw with respect to an established navigation frame. Unfortunately, the inertial sensor itself cannot be used to evaluate the visual odometry translation results due to relatively large inertial drift in the sensor measurements. As no independent measurements were available to evaluate translation with high precision, the truth reference was established by accurately measuring the actual paths taken during each flight.
A test flight was conducted traversing a rectangular indoor hallway loop. This test contained translation in multiple dimensions, large heading changes and a flight duration of four minutes. Moreover, this test allowed for evaluation of the eight-point algorithm and scale estimation method in the presence of rapid scene changes.
The attitude estimation results for this test are shown in Figure 11. Throughout data collection, the maximum separation between the inertial and vision-based attitude estimators for pitch, roll and yaw was 9°,19° and 14°, respectively. Upon comparison to many of the other conducted tests, the maximum attitude errors were larger. There are multiple reasons for this increase. First, the duration of this experiment was greater than that of previous experiments. Errors accumulate as a function as time due to integration of residual bias errors, so increasing flight duration will increase cumulative error.
Next, the looping path observed throughout this test caused the eight-point algorithm and scale estimation procedures to quickly adapt to differing scenery. Drastic scene changes (turning a corner) increase the difficulty of feature association between frames. This directly affects the procedures used for visual odometry in an adverse manner. Finally, there are situations in this flight where features are sparse. In general, a decrease in features will cause a decrease in the estimation capabilities of visual odometry.
Figure 12 shows the visual odometry path calculated for experiment 2. Here, the estimated length of each of the four straight legs of the rectangular loop matches to within 2 meters of the measured hallway lengths. This implies that the scale estimation technique is working reasonably well.
Figure 12. Visual odometry path determination while traveling around an indoor loop.
As for the estimated translational directionality produced by the eight-point algorithm, the first two legs of the loop never divert from the measured path by more than 2 meters; the third leg diverts by 5 meters. This is most likely due to a lack of well dispersed features in that specific hallway.
The cumulative error contained in the third linear leg of the loop also makes evaluation of the final leg difficult. However, if previous errors are removed, the final leg appears to match the measured path well. In total, the landing position calculated through visual odometry is 6.5 meters away from the measured end of the trial.
Integration Methodology
In cases where GPS measurements are available along with the visual odometry solution, the proposed method can extend the GPS/IMU integration mechanization. The structure of the referenced GPS/inertial integration consists of two filters: a dynamics filter that uses GPS carrier-phase measurements to estimate velocity and other IMU errors, and a position filter that uses the velocity output of the dynamics filter and GPS pseudoranges. The dynamics filter can be adapted and extended to include camera data within its mechanization.
The dynamics filter is a CKF designed to estimate the inertial error states: velocity error in the north-east-down (NED) coordinate reference frame, misorientation (including tilt error), gyro bias error, and specific force or accelerometer bias error. This yields a state vector. For a discussion of the state vector, download the supplemental PDF.
Results
To evaluate the proposed algorithms, data was collected through multiple flights of the hexacopter platform shown in Figure 2 through a structured indoor and outdoor environment including transitions between these two environments. The availability of GPS measurements in these environments ranged from fully denied, to substantially degraded, to enough observables for a full solution.
The results of one test flight are discussed in this section. Apart from the data collections with the hexacopter, truth reference maps were created for the indoor operational environment and used for evaluation of the described processes. The results of the full GPS/inertial/laser/camera integrated solution described in Figure 3 are shown in an NED frame in Figure 13.
Figure 13. Path compared to 2D reference map.
The truth reference of the environment, depicted in red (derived from a terrestrial laser scanner), is compared to the flight path obtained from the extended Kalman filter (EKF), displayed in blue. The estimated flight trajectory constantly remains within the hallway truth model, indicating sub-meter level performance. Furthermore, based on an extension of this work for environmental laser mapping produced from the EKF, combined with the accuracy of the map, it is further reinforced that sub-meter-level navigation performance is obtained.
During portions of the described data collection, there was enough visibility (>3 satellites) to calculate a GPS position. The availability of GPS measurements to the position estimation portion of the filter allowed for geo-referencing of the produced flight path and 3D map.
Figure 14 displays the geo-referenced continued flight path based on the integration filter superimposed on Google Earth on the left, while the standalone GPS solution based on pseudoranges only is plotted on the right. The geo-referenced path correctly displays the platform passing through Stocker Center, the Ohio University engineering building.
To demonstrate the contributions of the monocular camera to the above results, laser measurements were removed from the solution for a 20-second period where GPS was unavailable. During the 20-second removal of laser data, the system is forced to operate on integration between visual odometry measurements and the IMU. The cumulative effect caused by this situation can be observed in Figure 15. After coasting on an IMU/camera solution for 20 seconds, the path is subsequently altered by 3 meters, as opposed to the solution with all sensors.
Figure 15. Effect of losing GPS and lasers for 20 seconds.
To further emphasize the contribution of the visual odometry component, both the laser and camera were removed from the integration for the same 20-second period. During this time frame the EKF is forced to coast on calibrated inertial measurements. The effect of losing all secondary sensors for a 20-second period can be observed in Figure 16.
Figure 16. Effect of coasting on the IMU for 20 seconds.
During the forced sensor outage, a 45-meter cumulative difference is introduced between the path using all sensors and the path with denied sensors. Through comparison of the results shown in Figure 15 and Figure 16, the contribution of monocular camera data can be isolated.
When the EKF was forced to operate for 20 seconds using an IMU/camera solution, 3 meters of error were introduced. This is significantly smaller than the 45 meters of error observed when using only the inertial for the same period. Thus, the camera is shown to provide stability to the EKF when neither the laser nor GPS are available.
Conclusions
The visual odometry techniques produced reasonably good attitude estimation and are effective at constraining inertial drift when other sensors are not available. The inclusion of camera measurements to the discussed integrated solution resulted in increases in the accuracy, availability, continuity and reliability of the system.
Acknowledgment
The material in this article was first presented at the ION Pacific PNT conference in Hawaii, May 2017.
Manufacturers
The camera used aboard the UAV in these tests is a Point Grey Firefly MV and the IMU is an XSENS MTi. The GPS receiver is a NovAtel OEMStar with a corresponding NovAtel L1 patch antenna.
EVAN T. DILL is a research scientist in the Safety Critical Avionics Systems Branch at NASA Langley Research Center. He received his Ph.D. in electrical engineering from Ohio University.
STEVEN D. YOUNG is a senior research scientist at NASA with more than 30 years of experience in the related fields of safety assurance, avionics systems engineering and human-machine interaction.
MAARTEN UJIT DE HAAG is the Edmund K. Cheng Professor of Electrical Engineering and Computer Science and a Principal Investigator (PI) with the Avionics Engineering Center at Ohio University, where he earlier earned his Ph.D. in electrical engineering.
Following successful test flights, Insitu’s ScanEagle helps combat Oregon wildfire.
UAV company Insitu and Esri have successfully completed test flights on a new way to support firefighting efforts using software for firefighters and first responders.
The flights were held at the Warm Springs Federal Aviation Administration (FAA) Unmanned Aerial System (UAS) Test Range in Oregon. The test site is a Pan Pacific FAA UAS Test Site for commercial UAS testing. The national FAA test site program facilitates the UAS industry in meeting strict customer needs and qualifications.
Insitu is a wholly-owned subsidiary of The Boeing Company.
A week after successfully completing customer acceptance test flights, Insitu, which has more than one million operational UAS flight hours, deployed its INEXA Solutions professional aerial remote sensing teams to aid firefighters in suppressing the Eagle Creek fire in Oregon.
Onlookers watch the fire burn in the Columbia Gorge on Sept. 4. The fire is now contained. (Photo: U.S. Forest Service)
Collaborating with customers to identify business challenges, INEXA Solutions professionals use a continually expanding suite of capabilities such as INEXA Control (ground-based command and control), INEXA Cloud, INEXA manned and unmanned air vehicles including ScanEagle, and INEXA sensors and analytics to provide custom solutions and answers to mitigate business challenges from seabed to space.
Coordinating with the Oregon Department of Forestry and other governing entities, Insitu’s ScanEagle system provided optimal, near real-time data for firefighters and first responders, resulting in heightened emergency response efforts, increased situational awareness and safety, and supported planning and resource allocation.
Equipped with electro-optical (EO) for daylight and infrared (IR) video for nighttime flights, along with mid-wave sensors, the ScanEagle surveyed fire lines at night over the Eagle Creek wildfire, which had spread to nearly 49,000 acres throughout the Columbia River Gorge region.
The ScanEagle can supplement manned firefighting fleets by operating during dense smoke and at night, when manned aircraft typically cannot fly. Infrared camera technology can penetrate smoke and gather and disseminate georeferenced still images of points of interest. These images allow geographic information system (GIS) specialists to perform analysis using Esri’s ArcGIS software.
“Throughout the difficult Eagle Creek wildfire, our thoughts have been with our friends and neighbors impacted by this unfortunate event,” said Mark Bauman, vice president and co-general manager, Insitu Commercial. “We stand prepared to assist local authorities with ongoing operations in any way we can, and we extend our gratitude to all of those working hard to contain the fire.”
ScanEagle poised for launch at Eagle Creek, Oregon, fire.
As the sole aviation overwatch within the temporary flight restriction, the ScanEagle provided persistent nighttime oversight and monitored the progression of the fire. Insitu coordinated manned and unmanned aviation assets and through data collection, analysis and integration capabilities, produced near real-time georeferenced spatial data (maps tied to specific known locations).
In this way, incident commanders, firefighters, and first responders had data that delivered updated incident perimeter maps, identified spot fires, located fire lines and hotspots, and provided near real-time video feed and still images of critical infrastructure, historical structures and more.
“Prior to pursuing any new effort, we consider the reasons we exist as a company — we call it our ‘why,’ explains Jon Damush, Insitu’s chief growth officer. “Insitu’s ‘why’ is to pioneer and innovate in all that we do to positively impact people’s lives and change the course of history,” he continues. “This statement guides our actions and investments, and is precisely why we are doing the things we are doing to help those in need with our unique technologies and professional approach to aviation.”
