The University of Maryland (UMD) Unmanned Aircraft Systems (UAS) Test Site, along with and Shore Regional Health, conducted on Aug. 24 the state’s first civil unmanned aerial delivery of simulated medical cargo. Engineers from UMD flew a Talon 120LE fixed-wing aircraft across the Chesapeake Bay with saline solution simulating four vials of Epinephrine to demonstrate the key role that UAS can play in emergency situations.
First Responsders. “This is a major achievement for our test site and for the University of Maryland,” said Darryll Pines, dean of the School of Engineering. “What this flight demonstrates is the incredible potential that UAS have in assisting first responders in emergencies. As more of these aircraft enter the skies, demonstrations of their use in service to humanity will grow substantially.”
Weighing 22 pounds at take-off, the small UAS was hand launched from the shores of Flag Ponds Nature Park in Lusby, and landed at Ragged Island Private Airport in Cambridge, flying 12 miles over 28 minutes. The flight was autonomous with man-on-the-loop with ability to intercede.
The UAV was greeted by a security officer from Shore Regional Health who retrieved the package and transported it to the Shore Medical Center at Dorchester.
“We wanted to simulate a situation when weather, traffic or other disaster made more traditional means of transportation impossible. UAS are faster to deploy, less weather dependent and less expensive,” said Matthew Scassero, director of the UMD UAS Test Site.
Flight path as recorded by aircraft GPS. The loiter midway allowed confirmation of the radio monitoring/control signal handoff. Loiter will not be necessary for operational flights.(Image: UMD)
The test also helped Shore Regional Health explore new ways of providing access to medical care to rural areas, according to William Huffner, Shore’s chief medical officer. UAS technology has the potential to bring supplies not only to medical staff, but also directly to patients in isolated areas.
“In emergency situations, every second counts,” Scassero said. “Imagine being able to deploy insulin or another critical medication to someone in need by landing or dropping it right in their backyard.”
Talon UAV. The Talon 120LE is made of 7075 aircraft-grade aluminum, foam and composite materials. Scassero said that the team chose a Talon 120LE because of its “payload capacity, stability and reliability.” With an endurance of greater than two hours, its modular nose payload section and wing pods, it can carry payloads up to 2.5 pounds. The aircraft flies autonomously and lands on its belly.
Scassero said the use of UAS will be critical in future emergencies. “Using UAS for cargo will allow them to operate in tandem with manned aircraft to work together for these types of humanitarian missions and others, such as search and rescue,” he said.
Next Steps. Following this successfull test, the test site is looking at different operational control paradigms (suc as network or satellite), health IT cueing of the system, different vehicles for various applications, and different flight environments.
Integrated with images and dense distance measurements from a range camera using active illumination, inertial navigation produces real-time results on a tablet computer. Experiments demonstrate that the system provides good positioning and mapping performance in a range of indoor environments, including darkness and smoke.
Soldier with prototype system mounted on tactical vest.
Positioning and mapping abilities for indoor environments can speed search and rescue, keep firefighters from getting lost, and help a commander track soldiers searching a building. Accurate results from these environments increase personnel safety in unknown, dangerous environments and can also facilitate remote control of unmanned ground vehicles (UGVs).
A new iteration in the Chameleon family of positioning systems that we have developed, the Tiger Chameleon, combines an inertial measurement unit (IMU) with an active camera that measures distances using modulated laser light. This type of camera provides dense and accurate distance measurements, and has the added advantage of working well in darkness.
The main goal of the Chameleon systems is to provide soldiers and first responders with position information and approximate overview maps, preferably without affecting their operating procedure. The Tiger Chameleon does not require any infrastructure, such as visual markers or radio beacons. Positioning and mapping results are computed in real time based on data from the IMU and the active camera, and wirelessly transmitted to a visualization interface running on a tablet computer or smartphone.
Sensors and Hardware
In initial experiments, the Tiger Chameleon’s components are enclosed in a wooden box which can be mounted on a soldier’s tactical vest. FIGURE 1gives a schematic overview.
FIGURE 1. Schematic overview of how the components are connected.
Image Sensor. The image sensor includes two cameras: one high-resolution visual camera and one lower-resolution depth sensor. The latter uses a modulated near-infrared (NIR) light source and a special type of sensor to measure distance. Essentially, each pixel is divided into two halves, one of which integrates incoming light when the light source is turned on, while the other integrates light when it is turned off. If the light hitting a pixel has bounced off an object located close to the sensor, almost all light will be integrated by the first half of the pixel. If the object is moved further away, it will take longer for the light to return, and hence more light will be integrated by the second half of the pixel.
In addition to measuring the depth of the scene, the NIR camera provides an intensity image where the value of each pixel is determined by the total amount of light hitting the two pixel halves. The intensity image is similar to an ordinary visual image. Unlike an image from a passive camera, however, it is largely independent of ambient light, since it mostly measures reflected light from the illuminator. Hence, both the intensity and the depth images are available even in completely dark or obscured environments. Coming from the same sensor, the intensity and depth images are perfectly registered to each other. Images are produced at 30 Hz.
The high-resolution visual camera is not used by this prototype.
Inertial Sensor. The IMU provides calibrated measurements of acceleration and angular velocity at 400 Hz. The sensor itself does not perform any inertial navigation or attitude estimation. Since we fuse the inertial data with image-based features for navigation, however, the basic acceleration and angular velocity measurements are more useful in our application than pre-filtered position or orientation estimates. The sensor measures acceleration up to 5 g, and angular velocity up to 1,000 degrees/ second, since relatively high-dynamic motion is common in the intended application. The IMU also contains a magnetometer and a barometer, but these are not used since air pressure and magnetic fields are not always reliable for navigation indoors.
Algorithms
The positioning algorithm is based on EKF-SLAM, simultaneous localization and mapping (SLAM) implemented as an extended Kalman filter (EKF). The EKF fuses data from the IMU (three-dimensional accelerations and angular velocities) with image data, according to the uncertainties of the different types of data. It tracks the system state, composed of the position, velocity and orientation of the system, the IMU biases (slowly varying offsets in the acceleration and angular velocity measurements), and the positions of a number of landmarks in the images.
The landmarks are points observed in the images, which are used for navigation. These are chosen to be points which are recognizable, well-defined and stationary. This essentially means that it should be easy to recognize a landmark when it appears in a new image, and that the image coordinates of a landmark should be stable in both the horizontal and vertical directions. Thus, corner points are good candidates for landmarks, while line structures in an image are not. The world coordinates of a landmark should not change over time.
Theoretically, it would be possible to navigate using only IMU data. The system orientation would then be obtained by integrating the angular velocities, while the acceleration (after removing the effect of gravity) would be integrated to obtain the velocity, and double-integrated to obtain the position. Due to the high bias variability and noise of micro-electro-mechanical systems (MEMS) IMUs — the only type sufficiently small, lightweight and inexpensive for use by soldiers or firefighters — this only works for a few seconds before the accumulated error grows to an unacceptable level.
In theory, it would also be possible to navigate using only landmarks extracted from the image sequence. This, however, is also problematic in practice. If the system moves too rapidly, successive images may not share any landmarks at all. Additionally, no landmarks are found in featureless environments. This causes image-only navigation to fail in many realistic scenarios.
By fusing the inertial data with landmark observations, we alleviate most of these problems. While the IMU provides good short-term performance, the image data provides long- term stability. Hence, the IMU can overcome short periods with few or no landmarks, while the image data limits the error growth of the system (assuming that the periods without landmarks are not too long).
FIGURE 2. Overview of the algorithm.
Algorithm Flow. In the data fusion FIGURE 2, potential landmarks are found in an intensity image from the NIR camera, by identifying points that can be well localized. The potential landmarks are found by using an interest-point detector. The visual appearance of a point is represented using the SIFT descriptor. The distance to each potential landmark is determined by using the depth image. Potential landmarks are matched to landmarks that are already tracked by the system, based on a combination of their visual appearance and their coordinates (the spatial distance between the observed points and the predicted image coordinates of the tracked points, and the difference between the predicted and the observed distance to the points).
IMU data is used to predict where tracked points should appear. Pure inertial navigation is relatively accurate in the time interval between successive frames (approximately 30 images per second are processed). Observed points, which match tracked points, are used to update the estimated state (position, velocity and so on). Tracking is started for observed points, which do not match any tracked points unless a predetermined number (30 in the current implementation) of points are already tracked. Points that are tracked but not observed in a number of consecutive image pairs are removed from the point tracker.
Each depth image corresponds to a local point cloud, where points lie on the surfaces of objects in the scene observed by the camera. The intensity of each point can be obtained from the corresponding intensity image. Since the position and orientation of the system are estimated by the SLAM algorithm, these local point clouds can be transformed into a common coordinate system, thereby creating a large point cloud representing the entire environment along the trajectory of the system. This point cloud can either be used as a three dimensional model, or projected onto a horizontal plane for an overview map.
Implementation
To accurately predict where tracked landmarks will appear in a new image, it is important to synchronize the image stream and the inertial data. While the IMU handles this well (it can either trigger, or timestamp the trigger pulse from, a camera), synchronization is potentially difficult when working with consumer hardware such as the the one selected for this prototype. However, while the camera cannot be triggered externally, it does perform internal synchronization between the NIR and the visual cameras, which both run at 30 Hz. (It is unknown to us which camera triggers the other, or if there is a third piece of hardware which triggers both cameras.) Using an oscilloscope, we found that a pulse train at 30 Hz can be accessed at a solder point inside the camera.
This 30 Hz signal is active whenever the camera is powered. It is therefore not possible to synchronize the camera to the IMU using only this signal, since it only indicates that an arbitrary image was acquired at the time of each pulse. To synchronize the sensors, we need to know exactly when each specific image was acquired. Thus, a synchronization pulse, which is only transmitted when image acquisition is enabled, is needed. In addition to the 30-Hz pulse train, a signal that is high only when the cameras are active can also be found inside the camera. We perform a logical AND to combine these signals into the desired synchronization pulse.
We obtain the timestamp of each image from the IMU by connecting the combined synchronization signal to its synchronization input. There is a small delay between the first pulse in the combined signal and the time when the first image is acquired. This was measured by recording inertial data and images while first keeping the system stationary, and then rotating it quickly. Manual inspection of the image sequence and the angular velocities reveals that the delay is approximately 8 intra-frame intervals (0.267 s).
Software. The recording and analysis software are written in C++, and run in real time under Linux. It makes heavy use of multi-threading in order to optimize the computational performance of the algorithms.
The software is divided into two subsystems: one that communicates with the sensors, and one that performs the image analysis and SLAM computations. These subsystems communicate using the Robot Operating System (ROS). There is also a subsystem (or “node” in ROS terminology) for playing back previously recorded data. The playback node publishes the same type of messages as the sensor node, and these nodes can therefore be used interchangeably.
FIGURE 3. Overview of the analysis node. Blue boxes represent threads, while red boxes are queues. Purple boxes are other software components, and the orange and the red/green boxes represent the sensors. Orange, red and green lines represent data from the respective sensors. Black lines represent data based on more than one sensor.
FIGURE 3 shows how data flows through the analysis node. Queues buffer data between the different threads. Incoming images from the recording or playback node are initially put in the image queues for feature extraction. After feature extraction (where the depth image is used to determine the distance to each observed landmark), the resulting image features are stored in the potential landmarks queue. The features are then processed by the EKF-SLAM thread, which also reads from the IMU data queue. The resulting pose estimates (positions and orientations) are sent to the communication module, where they are transmitted to the visualization device. In parallel with this, images are also processed by the rectification and mapping threads, where they are paired with poses from the SLAM algorithm to create map data. The map data is also sent to the communication module. A number of efficient libraries exist for low-level image processing, and also for the linear algebra computations needed by the EKF. The following libraries are used:
VLFeat for finding landmarks and extracting SIFT features
OpenCV for image rectification
Armadillo for high-level math operations
ROS for communication between the subsystems
A modified version of IAI Kinect2 for ROS communication with the sensor
A slightly modified version of Libfreenect for low-level communication with the sensor.
Evaluation
The prototype has been evaluated in a number of experiments, in cooperation with soldiers and first responders. Three experiments are reported here. All were performed during soldier or smoke-diver training. Mapping results in the figures are presented along with estimated soldier and smoke-diver trajectories (bluegreen lines) from the positioning system. The results are evaluated based on the mapping, since the ground truth trajectories are unknown. Positioning is still evaluated implicitly, as inaccurate positioning would cause poor mapping performance such as double or skewed walls, since the mapping is based on estimated camera positions and orientations.
Searching for Biological Hazards. Collection of evidence at a crime scene is commonly performed using a camera, taking numerous pictures of the scene from different positions at different angles and distances. To keep track of all images and where they were captured is cumbersome work, which can easily be automated by integrating the camera with a positioning system.
Several variations of this experiment were performed, all in cooperation with soldiers from the Swedish Armed Forces National CBRN Defence Centre. The main task was to document different areas in the building, to be able to handle any encountered dangerous biological objects in a safe way, and to preserve evidence for later use. If a dangerous object is found, the mapping result from the system could be used to plan how the object should be taken care of and what exit path the soldiers could use to minimize the time being exposed to a dangerous environment.
FIGURE 4. Map estimated by the prototype positioning and mapping system while two soldiers searched for biological hazards. The trajectory is also shown. The grid spacing is two meters.
The photo above an image from the NIR sensor in the prototype system, captured in the building while two soldiers searched for biological hazards. A map estimated by the prototype system appears in FIGURE 4.
For evaluation of the mapping performance, a part of this building was also measured using a highly accurate scanning laser system. FIGURE 5 shows the point clouds from both the Tiger Chameleon (black) and the scanning laser (white). Locally (within a room), the errors are less than 5 centimeters. Over larger distances, heading drift causes larger errors, which are visible as slightly misaligned walls. In this case, there are no significant errors. (The structures that are only visible in the laser scanner data are parts of the ceiling and its supporting beams, which were never observed by the positioning system.)
FIGURE 5. Map produced by the Tiger Chameleon (black) overlaid on reference data (white).
Searching a Building. A group of eight soldiers searched and cleared a part of a building. This required them to move more quickly than in the first experiment, since the building could not be assumed to be safe. Additionally, one or several soldiers often appears in the field of view of the positioning system. We requested the soldier carrying the system to avoid walking too close behind another soldier, to avoid covering the entire field of view. Apart from this, the soldiers were asked to act as they normally would during the exercise.
FIGURE 6 Map estimated while eight soldiers cleared a building.
Positioning and mapping results are shown in FIGURE 6. The map is slightly distorted due to errors in the estimated heading, but still good enough to understand the building layout. The distortion is visible as double walls in the top part of the figure. We believe that most of the errors were caused by the system not being entirely stationary during the sensor initialization at the start of the experiment. No reference map is available for this location. We also performed experiments where the soldiers fired their weapons near the positioning system. This affects the IMU, severely degrading the measurements of acceleration and angular velocity. Hence, the positioning system was not able to present correct position or mapping estimates in these cases.
Smoke-Diver Searching a Building. The smoke-diver experiments indicated how the positioning system performs when used in a smoke-filled environment. During the experiments, the system was exposed to different levels of smoke.
FIGURE 7. Map estimated while a smoke-diver searched a (partly) smoke-filled building. The trajectory is also shown.
Positioning and mapping results from the experiment are shown in FIGURE 7, while FIGURE 8shows the result from the Tiger Chameleon (black) overlaid on reference data from the laser scanner (white). The estimated map is in good agreement with the reference map data, although a heading error affects some walls (bottom right).
All smoke densities affect the image quality from the NIR sensor, since the active illumination is reflected by the smoke particles. In light smoke, the main effect is that the smoke appears as spurious points in the point clouds, eventually ending up in the map, while positioning performance is not significantly affected. Most spurious points were removed by adding the constraint that points very close to the camera are not added to the map, although smoke still causes more interior points to be visible in Figure 7 than in the other maps. The effect on the positioning system increases with the smoke density. In thick smoke, most illumination is reflected, effectively rendering positioning impossible using this type of image sensor.
Discussion
FIGURE 8. Maps from the experiment in smoke. The map produced by the Tiger Chameleon (black) is overlaid on reference data (white).
The current positioning algorithm works well, but over longer experiments its position estimate slowly drifts away from the true position. This is caused by both the inertial sensors and the landmark-based positioning updating the current position estimate only relative to recent estimates. Since landmarks are discarded after not being observed for a short time, no loop closures ever occur. Saving landmarks could solve this in specific scenarios, where landmarks are re-observed after long times, but doing so would increase the computational complexity considerably. Additionally, for landmark-based loop closure to work well, the landmarks would need to be reobserved from approximately the same position, further limiting the scenarios where this could be expected to work well. Ongoing work aims instead at closing loops by recognizing the scene geometry, as represented by the point-cloud models.
When using active illumination, the mapping performance does not depend on texture on walls and other surfaces. This is an advantage compared to passive stereo cameras. Further, active illumination enables positioning and mapping in darkness.
Comparison to Other Systems. The prototype system was not constructed with the purpose of creating high accuracy models of small, detailed environments. Rather, the purpose was to enable soldiers to create approximate maps of entire buildings with minimal impact on their operating procedure. Detailed reconstruction is handled better by other systems, but these have the disadvantage of requiring the user to survey the environment systematically, adapting his or her work methods to the sensor.
SWaP-C and Hardware. Size, weight, power and cost (SWaP-C) are constant issues for soldier equipment. Ideally, the equipment should be disposable: cheap enough to throw away after being used just once, or fit into some type of disposable container.
Since this is a prototype system built for research and demonstration purposes, components with convenient electrical and programming interfaces have been selected. Therefore, the system is considerably larger than a final end-user product would be. In such a product, sensors and computation hardware with similar performance, but considerable smaller size, lower weight and lower power consumption, would be selected instead. Such components are commercially available.
Outdoor/Indoor Use. Though this positioning system is primarily designed for indoor use, seamless transition between indoor and outdoor environments is desired. Ongoing work aims at integrating a GPS receiver to achieve this. Adding GPS would also enable positioning in a georeferenced coordinate system; currently, all results are presented in a local coordinate system defined by the start position and orientation. This requires an algorithm for making robust decisions regarding when GPS measurements should be considered reliable.
Visualization Interface. The visualization interface currently runs on a tablet computer, and is therefore most useful to a commander or group leader who remotely tracks the soldier or firefighter. By adapting the interface to the smaller screen of a smartphone, it would be possible to also give the user access to position and map information.
An important advantage of automatic mapping, according to several users, is the ability to detect hidden spaces in buildings. In certain types of operations, the ability to document a building before leaving it is also considered valuable.
Real-Time Implementation. An interesting aspect of performing all computations in real time is that it effectively precludes tuning of algorithm parameters to individual data sets. When analyzing data offline, this is far too common, and typically overestimates the performance of the system or algorithms. All experiments reported in this article have been performed without any such parameter tuning.
Autonomous or Remote-Controlled Platforms. The system can also be used on an unmanned ground or
aerial vehicle (UGV or UAV). This could be suitable for searching buildings too dangerous to enter. The system would continuously distribute its position and mapping estimates while traveling through the building. This could provide soldiers and rescue teams with a preview of the unknown environment and a possibility to plan their operations in a safer way. Many robotics projects use ROS, which makes integration of the Tiger Chameleon relatively straightforward.
Firing of Weapons. During some experiments, we discovered that the measures of acceleration and angular velocity are affected by close-range gunfire. Relatively long segments of measurements are affected, which makes interpolation of missing values difficult. During these periods, it may be possible to disregard the inertial measurements, resorting to only image-based positioning.
Summary
This prototype system for indoor positioning and mapping, based on inertial navigation and distance measurements using active illumination, does not require any infrastructure or prior knowledge about the environment. The system has been designed for experiments and demonstration purposes, and has been shown to provide good performance in real time in a variety of different indoor environments when carried by potential end users.
Manufacturers
The Tiger Chameleon consists of a Microsoft Kinect v2, an Xsens MTi-10 IMU and a small computer with a mobile Intel i7 CPU.
Acknowledgments
The authors thank the soldiers from South Scania Regiment (P7), Sweden, who participated in the initial tests with the prototype. We also thank the smoke divers from Södertörn Fire Prevention Association, Sweden, for carrying the prototype in smoke-filled environments. Also, we really appreciated the feedback from the Swedish Armed Forces National CBRN Defence Centre during the prototype development. Finally, we acknowledge Hannes Ovrén at Linköping University, Sweden, for improving the Libfreenect library.
The material in this article is based on a technical paper presented at ION/IEEE PLANS 2016.
Horizontal indoor accuracy now, elusive z-axis by end of year
At their advent, mobile phones were conceived to be useful for when people were, well, mobile. And in 1996 when the U.S. Federal Communications Commission (FCC) first required that a handset’s location be sent to 911 dispatchers and meet accuracy performance standards, the FCC was understandably solely interested in calls made outdoors.
Indoor FCC rules
(rmnoa357 / Shutterstock.com)
In recognizing the pervasive use of mobile phones indoors and gains in location-determining technology, last year the FCC adopted new rules that establish accuracy requirements for indoor 911 calls.
The FCC didn’t stop there and tackled vertical positioning, ordering that within six years, the elusive z-axis, or altitude, be added to requirements and meet accuracy standards in cases when there is no dispatchable location. The z-axis is critical in finding a person in a building of more than one story, whether a high-rise apartment building in Brooklyn or a three-story dormitory at a university.
This spring, a testbed for verifying location technologies began operations. The FCC required that nationwide wireless providers create an independently administered and openly transparent test bed to verify location technologies used in meeting the accuracy requirements. CTIA, the trade association for the U.S. wireless communications industry, established the 9-1-1 Location Technologies Test Bed as an independent company.
Testing is designed and administered by ATIS, an industry standards association. The testbed regions are located in metropolitan Atlanta and San Francisco and cover a wide range of building types and terrain.
Indoor testing will be performed in 20 buildings within each test region, spanning four morphology types (dense-urban, urban, suburban and rural). Test bed administrators will not divulge the technologies being tested.
No Silver Bullet. The FCC acknowledges that there won’t be one silver bullet location technology, one size fits all that will be the best location solution in all situations.
In the order released on Feb. 3, 2015, the FCC writes, “To be sure, no single technological approach will solve the challenge of indoor location, and no solution can be implemented overnight. The requirements we adopt are technically feasible and technologically neutral, so that providers can choose the most effective solutions from a range of options.
“In addition, our requirements allow sufficient time for development of applicable standards, establishment of testing mechanisms, and deployment of new location technology in both handsets and networks… Clear and measurable timelines and benchmarks for all stakeholders are essential to drive the improvements that the public reasonably expects to see in 911 location performance.”
The 9-1-1 Location Technologies Test Bed has begun indoor testing of currently deployed horizontal location technologies, and its results will be used as part of location accuracy compliance reporting to meet FCC benchmarks.
Toward the end of this year, location technology vendors will use the Test Bed to test near-term emerging horizontal and vertical location technologies, such as z-axis, that are not currently deployed by the nationwide wireless carriers.
JANICE PARTYKA is GPS World’s contributing editor for wireless. She is principal at JGP Services and provides strategy and marketing consulting to the mobile industry. She reported on a previous round of tests, the 2013 FCC-chartered Communications Security, Reliability and Interoperability Council (CSRIC) trials of NextNav, Qualcomm and Polaris technologies. See gpsworld.com/indoor-trial-results-next-fcc-chief/.
Last month I raised my anxiety level by writing about a revenant threat from terrorist-initiated biological attacks.
The same concerns were also cited by Director of National Intelligence James Clapper during recent Congressional testimony. These “poor man’s nukes” could potentially be more devastating than 9/11 and reach into every community and even our own homes. Additionally, the threats are not easy to ferret out and just as difficult to stop in our very complex and interconnected world.
From bioterrorism to natural disaster emergency management, predictive analytics used with geospatial tools and big data is proving to be a powerful new intelligence tool that may help counter global threats.
TransVoyant Predictions
TransVoyant CEO Dennis Groseclose.
Last year there was a lot of buzz at GEOINT surrounding a relatively new company in this field called TransVoyant. Several weeks ago, I visited TransVoyant’s Alexandria, Virginia, headquarters to learn more about their capabilities first hand. I was fortunate to be able to speak with TransVoyant CEO Dennis Groseclose, an Air Force Academy graduate who, with Tim Fleischer, a Naval Academy graduate and successful entrepreneur (Radian, PD Systems), co-founded TransVoyant.
Previously, Dennis led industrial base optimization restructure for the $37 billion dollar unmanned space launch program for the U.S. Air Force; directed and implemented Worldwide Supply Chain Optimization for IBM; and served as vice president at Lockheed Martin. These experiences built his expertise to solve complex supply chain and global risk management problems using advanced analytical intelligence. In 2011, Dennis and Tim put their collective experience together to form TransVoyant, a company that specializes in creating live and predictive insights from real-time big data.
The Internet of Things (IoT) has been a key component of their operation. In the mid-80s, connected remote sensors numbered in the thousands. In 2016 that number is expected to reach 6 billion connected “things” worldwide with estimates of 30 billion by 2020.
TransVoyant collects, cleanses and analyzes over 200,000 external events around the world every minute (such as severe weather, natural disasters, labor strikes, inventory locations, news, terrorism incidents, disease outbreaks and energy prices) from real-time IoT data sources such as sensors, radar, GPS, satellites, smartphones and meters. It then continuously applies advanced data scientist-crafted analytics to these data streams to assess important current and future behaviors, impacts, correlations, patterns and exceptions that deliver live and predictive insights ranging from forecasts of port disruptions and precise shipment arrival times to forecasts of economic flows to real-time and predicted threats to people and assets. The resulting insights — provided via cloud services, system API connections, email and mobile applications — improve mission-critical decision making.
The geospatial grid connection
This was all sounding like science fiction and black magic until an “aha moment” for me, as Dennis explained how they use a “multi-dimensional grid cell mathematics” based data structure to apply complex algorithms to real-world data and events. This put the very complex process of continuous real-time machine analysis that “understands” normal and abnormal behavior, both current and future, into something that was familiar to me.
Decades ago, I used the first release of ArcINFO GRID, now called ArcGIS Spatial Analyst, to complete my master’s thesis. For those of you that haven’t used a grid-cell-based GIS, let me highlight the differences between that and traditional GIS software.
Traditional GIS software describes our world as points, lines or polygons with topology describing the mathematical spatial relationship between each geographic element and its linked record in a database. This topological model is somewhat cumbersome and slow because the shapes and topological relationships are complex.
Grid: David Kidner, Mark Dorey & Derek Smith, University of Glamorgan, Wales, U.K. CF37 1DL
The other kind of GIS is a grid cell or raster-based GIS. The data model is significantly simpler because — unlike a traditional GIS of points, lines and polygons — the grid-based GIS world is broken up into simple uniform grid cells.
The big advantage is that the data structure and tools lend themselves to very fast processing. Almost any mathematical formula can be used to operate on the individual or collective grid cells. Most grid-based systems use predefined mathematical operations such as shortest path analysis, interpolation including Kriging or very complex formulas using map algebra.
So, very similar to the famous Napoleon Hill quote, “Whatever the mind can conceive… it can achieve.” With a grid cell GIS, if an analyst can think of a way to describe an analytical process and predictive results as a mathematic expression or formula, it can be done very quickly in the grid cell environment. (See two previous columns for more in-depth information — “GRID Cell Modeling” and “Topology is not Topography”.)
So what does grid cell GIS look like in action?
Evacuations during a flood.
Proactive Emergency Response
In my discussions with Dennis, a TransVoyant customer segment that caught my attention was support of first responders. Emergency responders are using TransVoyant to help with very early disaster response. One specific example is evacuation of invalid patients before a flooding disaster becomes life threatening.
A hospital evacuation.
Using TransVoyant’s analytics on an extensive network of satellite imagery, 911 and 311 calls, water-stage sensors, street closures, weather forecasts, registries and more, responders can predict areas that are at high risk for flooding hours before flood waters rise. Among other essential emergency management actions, these early warnings provide emergency responders with the ability to identify specific neighborhoods and homes that have disabled residents who can be easily evacuated, increasing the safety and efficiency of their operations.
Here is a screen capture of TransVoyant Continuous Decision Intelligence (CDI) predicting a flood event.
TransVoyant Continuous Decision Intelligence (CDI) predicts a flood.
Other Clients
TransVoyant’s live and predictive insight solutions have attracted customers ranging from large multinational corporations to National Security and Intelligence agencies.
I know that one hears echoes of Minority Report when reviewing the tools and capabilities of TransVoyant. However, given the serious threat we face for a situation far worse than 9/11, I have no reservations about using open-source data aggregation and brilliant analytics that correlate and uncover patterns of life and global anomalies to divine a threat.
So, will predictive analytics be the tool that stops a bio terrorist or saves lives in critical emergency situations? I don’t know, but the potential threat is too grave not try every tool, including continuous precognition, in our collective toolbox.
TransVoyant will be an exhibitor at GEOINT 2016 this month, so stop by and learn more.
Since we are approaching Mother’s Day and Memorial Day, I’d like to call your attention to my May 2014 column. It’s the best column I ever wrote.
When I entered the civilian part of my GIS career as the GIS manager for the Atlanta Regional Commission, I tried to get first responders interested in GIS. Of course, in the early ’90s we were happy to be able to accurately draw points, lines and polygons on a piece of paper. Soon we had the luxury of ortho imagery as a backdrop for our GIS data, but I still couldn’t build a lot of enthusiasm among those first responders.
That changed completely when we started using metric oblique imagery provided by Pictometry. I realized that since we live in an oblique/3D world many non-GIS users had real difficulty visualizing objects or locations using two-dimension visualizations such as drawings, blueprints, maps or even ortho imagery.
By contrast, oblique views made visualization much easier for the vast majority of non-GIS users, and use of oblique imagery coupled with GIS tools exploded. Since then, many of us have been searching for faster, easier and cheaper ways to collect oblique imagery and video, and build 3D models.
For more than a decade, major defense contractors developed leading-edge systems to capture and exploit aerial imagery and video. Although effective, as one would expect of new custom technology, the systems were very expensive and out of reach for most local government agencies. Remote GeoSystems seems to have developed a system that leverages current technology to provide capabilities that may address some of those needs at a reasonable price.
Remote GeoSystems is in the business of capturing, displaying and managing “georeferenced” video and imagery. The company has designed and built high-end geospatial video recording systems for full motion video (FMV) and GIS mapping software primarily aimed at regulatory compliance of energy corridors, grids and critical infrastructure inspection applications.
Fortunately, my UAV is a DJI Inspire 1. I chose the Inspire because of its reputation, and because it seems to be the best combination of features needed for first-responder work at a prosumer price (about $3,500). The Inspire can record up to 4K video/12-mp stills, has a 94-degree field of view so there is no wide angle “fish-eye” distortion typical of an action camera, and has “Lightbridge” technology that permits positive control up to 3 miles and the ability to stream live 720p video (now 1080p) back to the ground controller.
The controller can feed large-screen video for command center group viewing via an HDMI output. Most important, the Inspire records GPS position data and altitude along with the video/imagery stream. (The DJI Phantom 3 Pro is a cheaper alternative that also records telemetry data, but if one upgrades to a 4K camera and the Lightbridge transmitter/receiver, the price approaches the integrated Inspire 1 price.)
An .srt file.
Since I’m always leery of marketing pieces and company demos, I wanted to try the system myself, and Remote Geo was happy to oblige. My first hands-on test was very satisfying. The LineVision software downloaded, unpacked and loaded quickly with no problems. I then recorded some aerial video of our condo building on Lake Guntersville near Huntsville, Alabama. I chose this building because it was convenient, safe to fly and a multi-story building in the open.
In addition to recording the video, one needs to turn on the DJI Inspire metadata recording to generate the .srt file. This is done in the DJI application “General Settings/Camera” by toggling “Video Caption” on. The .srt file was initially designed to provide altitude and location data as on-screen captions, but the data can be used as needed for other purposes.
When done with the flight and recording, transfer the video file and .srt file to your computer. Make sure the video file .mov/.mp4 and .srt file are in the same folder. Open LineVision and you will see an ArcGIS window. From the pull-down menu, load the video and you will instantly see the video play in a separate window with red position dots on the ArcMap view. As the video plays, the dot associated with the location of the UAV will turn yellow. If you click on any dot, the video will jump to that location/position on the video.
Here are screen captures of LineVision showing the ArcGIS view of an ortho image with red dots illustrating the path of the UAV:
One advantage of LineVision for first responders is that it is a complete package with ArcGIS embedded, all for a price well below $1,500. There is no need for a separate ArcMap license. Additionally, although LineVision Esri ArcGIS can display GIS data from online sources, if you have GIS data for your location loaded on your computer the system will operate in a disconnected remote environment. These sample screengrabs don’t do the system and video justice, since I recorded at 1080p rather than 4K. My laptop, this website and the reader’s playback equipment limit accurate playback of 4K content, so I did my work at 1080p.
I can envision a disaster-response scenario where the response team arrives on site, launches a UAV, and starts recording the scene. The captured video could then be loaded, viewed, indexed and cataloged with GIS data overlays on a laptop all in a matter of minutes, even in a disconnected environment. Hours, days or months later, finding the right video clip for analysis or forensics should be significantly easier and faster.
With the explosion of UAV hardware and software, it’s going to be an exciting year as new smaller, cheaper and more capable systems hit the market. Remote GeoSystems is working with UAV manufacturers to make LineVision capability available for many of the newcomers.
Leveraging UAV and LineVision capability, Skyline has worked with Remote GeoSystems to bring yet another capability: rapid 3D model creation. Taking appropriate geo-located frames of the video, Skyline uses its PhotoMesh software to build fully metric 3D models in short order. The full capability of this system and its 3D viewer TerraExplorer is so extensive that I will cover it in a future column, after this month’s ESRI Federal Users’ Conference. If you see me at the UC Feb. 24-25, please stop me and say hello.
Last summer at the Space and Missile Defense Symposium, GEO Huntsville held its annual GEOINT workshop including a keynote by NGA (National Geospatial-intelligence Agency) Deputy Director Sue Gordon. One of the sessions, presented by Mike Botts, focused on the OpenSensorHub and related information published on GitHub.
His topic: clearing the path for use of geospatial-capable devices via the Internet, thus preventing a geospatial Tower of Babel.
In the mid-80s, I purchased my first personal computer from Sharper Image, a 286 with a monochrome monitor. The PC was not bad for its time, and I learned a lot about personal computing, but hooking up a dot-matrix printer at the time was a nightmare. There were numerous types of printer cables — 25-pin parallel, 36-pin Centronics, 15-pin, etc. Additionally, some printers needed changes to the pin configurations, so nothing about the process was easy.
Then, after the mechanical connections were made, proper drivers had to be loaded, not to mention operating system and software configuration. Today, you simply plug in a USB cable or go wireless and are off and running thanks to “plug and play.” However, plug and play is only common in popular mass-market devices such as printers, scanners and cameras. Most other devices, even commercial consumer devices, can still present maddening connection challenges.
One example: About five months ago, I tested more than a dozen different Internet video security cameras for a special project. All the cameras I tested touted quick and easy connection. Some were quite nice, while others were installation torture — I returned those after a few days.
One well-known consumer brand was especially bad. I spent more than three hours with hard-to-understand tech support in India, and after countless different IP configurations and tests, I gave up. I decided that my remaining life is too short to waste that much time on a poorly designed camera system.
(By the way, the FLIR FX and Netgear Arlo were my top choices. Both connected fast and easy, both have especially nice cloud applications and both are wireless, including power. The FLIR is rechargeable, but the battery life of the Arlo seems remarkable, although some reviewers differ, especially outdoors and in freezing weather. In my test, after three months of flawless operation indoors, the Arlo is still on the original set of batteries at 60 percent, so it gets my top nod.)
OpenSensorHub
What is OpenSensorHub, and what are they doing to help achieve universal plug and play? By their own definition:
“OpenSensorHub is a license free, open source software platform for geospatial (FOSS4G) sensors that allows you to easily, rapidly and affordably network sensors into a seamless SensorWeb of real-time, location-aware, interoperable, web accessible services. With OpenSensorHub, these OGC compliant SensorWebs can be enabled across all manner of space-based, airborne, mobile, in situ and terrestrial remote sensors — including your basic mobile device. OpenSensorHub finally makes it possible to integrate location-aware sensors into the geospatial mainstream.”
(FOSS4G — Free and Open Source Software for Geospatial — is an annual recurring global event hosted by OSGeo growing out of the GRASS and MapServer communities. OSGeo — Open Source Geospatial Foundation — promotes open source software and resources. OGC — Open Geospatial Consortium — promotes open geospatial standards for both open source and proprietary software.)
The OpenSensorHub evolved from the early work of Mike Botts of Botts Innovative Research and Alex Robin of Sensia Software for NASA. They very laboriously designed and developed systems and software to connect sensors and actuators into an interoperable and integrated environment. They also realized that this connectivity and integration process had to become streamlined and not a custom programming effort every time for every device. Thus was born the idea of Sensor Model Language (SensorML) and, thanks to NASA funding in 1999, it became a reality.
Over the years, many scientists and engineers worked to develop connectivity for devices that could be queried and controlled through the Internet, called the Internet of Things (IoT). However, a key missing element of IoT was location awareness, so in 2000, SensorML was brought to the Open Geospatial Consortium (OGC) and served as a catalyst for the creation of a suite of open standards to support location-enabled discovery, access and tasking of sensors through web services and XML encodings. They named it the OGC Sensor Web Enablement (SWE) standards, or SWE for short.
The SWE standards, now in version 2.0, have been adopted worldwide supporting scientists, emergency responders and the military. Although SWE opened the door to geospatial integration, much work still remains to achieve true plug-and-play connectivity of thousands of devices. In my mind, SWE is standardizing communication protocols between sensor and actuator devices, much like USB standardized interactions between disparate devices.
However, what really enables us to plug in a USB cable and have instant and effortless communication between various devices, is the software and hardware that implement the USB standard protocols. This, in essence, is the focus of the OpenSensorHub community, to provide open software and hardware that fully implement the SWE vision and enable us to have effortless interaction between IoT devices.
This is also where the OpenSensorHub community needs your help. In addition to helping improve the significant capabilities of the OpenSensorHub Core, the OpenSensorHub community is looking for those interested in deploying sensors and in developing adaptors and adaptor technologies for adding new sensors, actuators, and processes.
If you’d like to learn more about the technology and ways that you can contribute, check out the OpenSensorHub website or contact the team at [email protected].
In this screen capture of the what3words app, the pointer is on mouse’s head at the Magic Kingdom. That grid cell is named “perform.heckle.comfortable” and will not change.
Early this year, I wrote a short column about what3words, one of the exhibitors at the Esri Federal GIS Conference. Since then, I’ve run into a fair number of geospatial professionals who hadn’t heard about what3words. This month, I’m doing a deeper dive on it because I believe it will become part of our daily lives in just a few short years.
What is what3words?
what3words is a global location system using tessellated grid squares of the entire Earth. Each grid cell is roughly 3 meters by 3 meters, and each cell is uniquely named using a simple three-word combination such as “fork.lamp.book.” On initial consideration, one would think, “So what?” — until you understand the ramifications.
First, this has already been done. More than 57 trillion 3-meter squares have been named using only 40,000 words.
Second, the system is non-hierarchal, and the cells have no adjacent relationship, so minor errors are dramatically obvious.
Third, unlike GPS lat/long, the United States National Grid (USNG), the Military Grid Reference System (MGRS) or even street addresses, the three-word combinations are easy to remember and not easily misunderstood.
Fourth, the system is not just a 57-trillion record database; it’s a compact app (10 mb) that accurately generates the same unique name for each unique location with identifiers that are locked in concrete.
The what3words website has more information and a well-done video overview.
How did it come to be?
Surprisingly, what3words was developed not by a geospatial analyst, but by a musician who got tired of driving around trying to find the correct hotel loading dock or concert venue entrance using an address or verbal directions. Even GPS coordinates didn’t help, since it was easy to miskey numbers or misunderstand voice-relayed numbers. As a result, he and his team built an app that is easy to use, memorable and not error prone.
Early radio analogy
The system is so easy to understand that non-technical users can quickly adopt it. I believe that it will greatly speed communications, minimize mistakes, and reduce wasted time and mileage. To me, a good analogy is the World War I development of the phonetic alphabet.
In the early days of radio, voice communications were difficult and error-prone because of static, noise and garbled transmissions. To prevent mistakes, the military adopted a fixed list of words to help with aural identification of individual letters. The words were used for transmission of critical information such as map coordinates or to spell out words. (Alpha, Bravo, Charlie, Delta, etc.) A similar mind/ear relationship occurs with what3words. Here is a well-written technical appraisal and amplification by Prof. Robert Barr.
Possible uses
Military
The implications for the military could be significant. When I served on a destroyer, one of my duties was Gunnery Liaison Officer, providing naval gunfire support for troops in battle locations. The 5-inch gunfire was called in by concealed spotters in the battle space. The coordination and conversion between the spotter location, the spotter’s point of view, and our offshore position and line of fire required significant calculations and diligence, because friendly fire was always a concern. Current developments in GPS and laser technology have helped significantly, but friendly fire mistakes from guns, missiles and bombing are still a constant concern. The use of what3words could be a simple and quick way to double check and prevent targeting friendly locations.
Another issue that was a problem for some military bases was addressing, or E911. Some bases had buildings identified by numbers corresponding to the sequence of construction rather than street addresses, so building 245 might be next to building 1842 and next to building 38 (I’m not sure if this is still an issue). With what3words, help could be directed to exact building entrances or to exact locations in remote parts of a base.
Disaster response would also benefit. In many disasters such as tornados or hurricanes, street signs and building were obliterated. What3words would provide “addressing” for relief supply drops and other needs.
The location of the helo deck on the battleship Wisconsin in Norfolk, Virginia, is identified as “chew.sketch.hardly”.
First Responders
Whether it be an air crash needing remote mountain rescue, a farm accident in a rural area, a capsized boat at sea, or a heart-attack victim in a shopping mall or home, response could be significantly faster with less chance for error. Even in urban areas, there are frequent stories of delayed medical aid because E911 street databases were not correct or updated with new construction. what3words provides complete location coverage and would serve as an easy and effective double check for street addresses.
Government and NGO activity
Some of you may be familiar with U.S. Census Bureau TIGER files and LUCA (Local Update of Census Addresses). I still can’t wrap my head around why census workers have to keep posted house numbers and street names confidential. Perhaps using what3words could provide a simpler, unclassified way to direct census workers. Additionally, many actions that currently use GPS may be better served with 3m grid locations, such as agricultural or environmental data collection.
Business
Mundane activities such as materials delivery to unaddressed construction sites or package delivery to homes and businesses will be more efficient. (Rumor has it that a prominent package delivery service is testing what3words.) Utility companies could locate cut-off valves, meters and other assets within 7 feet of their actual location. Meeting friends, getting an Uber pick up, or even having a pizza delivered to a specific bleacher location at a Little League game would become easy.
Second and third world
There are complex issues regarding the World Bank and economic development. To qualify for major economic development loans, countries have to demonstrate that they have viable property ownership and taxation system in place so loans can ultimately be repaid. We take our tax parcel system for granted, but may third-world countries don’t even have consistent and comprehensive street names and addresses. what3words can provide “addresses,” which could lead to more comprehensive parcel identification.
On an even more basic level, the majority of citizens in the world don’t have an address for simple deliveries. When I was in a rural part of Haiti, just getting some simple wood screws was an all-day trip and ordeal. I learned to really appreciate being able to take a quick run to Home Depot or get two-day deliveries at my front door from Amazon. Those “luxuries” don’t exist in many parts of the world, and their lack really cripples those trying to start or run a business. what3words gives everyone an “address.”
Try it
Don’t take my word for it; try it yourself. Download the app on your smartphone (I’m using an iPhone, so others may be slightly different). Launching the app will display a map with your location and its what3words name. Click the “eye” to view an ortho image rather than a vector map.
If the padlock is locked, unlock it and you can move the map to different locations showing different what3words names. If you are sent a what3words location, you can click on the magnifying glass and type in the three words. The app will prompt “Near Me” or “Anywhere.” If there is no match near you, it will show possible options that come close by looking at alternate spellings or words. If you click “Anywhere” it will search the entire world for a match.
Once it takes you to the location, you can use Maps or Google Maps to get directions.
Other points
what3words has been adopted by many geospatial firms, including Esri. Available online or offline, anywhere in the world, the what3words locator can be available to the GIS team or customers across the entire ArcGIS platform. Since what3words is grid-cell layer, it may be possible to do map algebra operations on the cells in Spatial Analyst. I’m not sure there would be a benefit to that other than not needing to transform the list of affected cells.
what3words is available in several languages. The words are not simple translations, but developed for each language. Although the what3words team carefully scrubbed the words used to avoid offensive terms, I hope what3words doesn’t have to deal with lawsuits from individuals unhappy with the three-word identifiers of their location.
Conclusion
I predict that within a few years, our business cards will also include a what3words address. Simply put, I believe that what3words may prove to be one of the most significant geospatial advancements since Jack Dangermond spatially linked points, lines and polygons to a relational database.
what3words is going to save time, money and, most important, it’s going to save lives.
P.S. If you read my March column reviewing Peter Zeihan’s book The Accidental Superpower, you may remember the importance Peter placed on 3D printing affecting the geo-politics of shipping manufactured goods from China. If you haven’t seen the new CLIP technology 3D printers, you need to view this TED video to see how far the technology has progressed.
When I was the GIS manager of the Atlanta Regional Commission, the most rewarding and important work we did was geospatial support for our first responders. The culmination of this effort was creation of a portable GIS that we could set up in the field on short notice anywhere in the region to provide situational awareness for first responders.
The system consisted of two laptops, external hard drives, a HP “E”-size plotter, foam-board laminator and an LCD projector — all housed in a portable tent. We used ArcInfo and ArcView to build and overlay vector data on ortho/oblique aerial imagery to aid visualization.
We found that police and firefighters especially liked our large laminated plots of imagery overlaid with street data, because the aerial images were easy to understand and the GIS data provided needed location references. The hard-copy plots required no computer and could be marked up with grease pencils.
Helping in our small way, we provided the same kind of large plots of New Orleans to the Louisiana National Guard days after Hurricane Katrina hit. We later learned that the plots were used by National Guard headquarters to keep track of search-and-rescue efforts by marking up neighborhood blocks with grease pencils and recording search results. They crossed off buildings that had been searched and recorded urban rescue information such as who did the search, and the date and number of live or deceased bodies found. The hard-copy plots were a low-tech embodiment of higher tech GIS data and imagery.
Nine years later, the National Geospatial-Intelligence Agency (NGA) developed a similar but higher tech and more robust system called GeoQ.
GeoQ: Geographic Work Queueing and Tasking System
GeoQ is an open-source geographic tasking and management system that facilitates collection and display of diverse geographic and geographically tagged data across large areas to provide situational awareness for all involved. As needed, the large areas can be broken down into small grid squares and assigned to teams or team members for detailed analysis or tasking.
The system is designed to be very transparent so all involved can view the workflow and assist as needed, while avoiding duplication of effort. This NGA video is a well done and rapid overview of GeoQ.
GeoQ software was developed by NGA and the MITRE Corporation to leverage NGA tools and data to the benefit of Homeland Security personnel. In 2013, the leadership at NGA made a gutsy decision to share some of their unclassified geospatial tools with the nation’s first responders through GitHub, an open-source software developers’ online collaboration environment. With more than 2 million participating programmers, GitHub hosts more software source code than any other single service in the world.
GeoQ was the first NGA product shared through GitHub, and was in keeping with a change in philosophy at NGA to take advantage of feedback and improvements generated by the huge and diverse talent pool available through GitHub. NGA was the first intelligence agency to share some of its work in this open-source environment, and the results have been extremely beneficial to all involved.
The Huntsville Connection
With more than 70 geospatial firms and agencies, Huntsville, Ala., has always been an early adopter of geospatial technology. Several years ago it was again a Huntsville team that developed a first-ever Google Earth Enterprise-based emergency response system called Virtual Alabama. The system was so effective that eight other states adopted the model. Work was underway to build a national version when Google announced the phasing out of Google Earth Enterprise. We now know that Google was not motivated to build authoritative geospatial systems, but was focused on building products and services that attracted customers so it could accomplish its primary business of selling advertising.
Fortunately, the work of the Virtual Alabama team was not wasted. Team members became experts regarding first-responder operations and their unique situational awareness requirements. They learned that first responders needed much more than just a GIS. The best analogy I can think of is that one could use PowerPoint as a word processor, but that wouldn’t be a very efficient system. Likewise, MS Word could be used for presentations, but not as elegantly as PowerPoint. The same holds true for rapid dissemination, communication and perception of a common operational picture. Geospatial tools and analysis are part of situational awareness, but the work flow and many components are different, not necessarily spatial and need to be assembled and processed at their own pace.
The timing was almost perfect, since GeoQ was released as Google Earth Enterprise was being phased out. Huntsville again seized the opportunity to build on its experience, and GEOHuntsville became a prime GeoQ testbed. I recently met with Chris Johnson, one of the early Virtual Alabama leaders and president of A Visual Edge, Inc., a Huntsville geospatial firm. She demonstrated GeoQ and Huntsville’s role in advancing the technology.
GitHub has a very thorough description of GeoHuntsville, a non-profit 501c6, and its role with NGA to test and share lessons learned through a “Blueprint for Safety” pilot project involving other cities to improve rapid disaster response. The sharing of lessons learned, code sets and documentation through the multi-city collaboration is called “Exemplar City.”
Another aspect of the Blueprint for Safety is support of rapid sensor deployment in support of first responders through common standards. Sort of a “plug-and-play” for complex devices. (See the GeoQ projects page and the OpenSensorHub.)
There is quite an extensive collection of material on GitHub regarding GeoQ and other NGA, projects including support of FEMA and GeoQ technical specifications. NGA doesn’t directly support these efforts, with legal language such as “NGA assumes no responsibility for the use of the software by any parties, and makes no guarantees, expressed or implied, about the software quality, reliability, or any other characteristic.” Still, NGA is behind the creation of the software and is working on other tools and support that will expand the capabilities. Participation of both GeoHuntsville and the Federal Emergency Management Agency (FEMA) bodes well for future use, since I don’t envision the same situation we ran into with Google and Virtual Alabama/USA.
Update on What3Words
In February, I wrote about what3words. The simple what3words system is now available as a locator, accessible via the Esri ArcGIS platform.
Two new apps developed at Texas A&M University-Corpus Christi use social media to help police officers, news stations, and the public navigate the many incidents and minor emergencies that may occur on a daily basis.
Richard Smith, creator of the two emergency response apps, is collaborating with Michelle Maresh-Fuehrer, assistant professor of Communication at A&M-Corpus Christi, to identify how the apps could aid first responders during emergencies.
“With the combination of SituMap and PhotoSorter, the public can be encouraged to submit photos and videos that may be helpful during an investigation,” said Smith, assistant professor of Geographic Information Science and Geospatial Surveying Engineering at the Island University. “For example, during an active shooter event, photos and videos of the suspect or their location can be taken with a cell phone and easily sent to responders. This could drastically improve response time and ultimately save lives.”
Photo credit: Texas A&M
Smith developed the mapping applications to provide a way for first responders to rapidly, and easily, receive and map information so they could have a more comprehensive awareness of emergency situations. Maresh-Fuehrer is working on extending the use of Smith’s social media mapping applications to enhance communication before, during, and after a crisis.
“A crisis event is typically a time of high stress and increased uncertainty for organizations and responders,” said Maresh-Fuehrer, who studies crisis communication strategies. “The applications developed by Dr. Smith have several features that allow for more informed and efficient crisis response.”
SituMap acts as a tablet-like digital command center that shows officers maps of the crisis area. With the touch of a finger the table-size display can be zoomed, rotated and drawn on. Like a personalized version of Google Maps, officers can search for locations and measure distances. But it goes further than Google Maps. A pin can be created in the application that could represent a person, police car, or groups of people. The pin can be strategically positioned around the area and directions can then be relayed to officers at the emergency location.
“An organization’s crisis team, along with emergency responders, can use SituMap to identify where people should be during a specific crisis,” said Maresh-Fuehrer. “With this application, responders can even view floor plans. This could help to identify safe locations such as fire exits and stairwells.”
PhotoSorter works in tandem with SituMap by allowing emergency responders, crisis planners and community members to share pictures or video of the crisis. Emergency responders can then upload the photos and video into SituMap to help in important decision-making situations.
SituMap and PhotoSorter were designed and developed at the Island University by Smith. The University Police Department is currently using a beta version of SituMap in training sessions.
In today’s digital world, people all over the globe can be connected through social media and, with the touch of a button, information about a major accident can be shared worldwide. With SituMap, important responders, as well as the community, can see real-time information on traffic congestion, roadblocks, and closed roads, which will aid in faster response times. The app also has a weather feature built into it which could be used during severe weather events such as a hurricane.
Ethertronics has unveiled EtherHelix GPS, a small, stand-alone, Right Hand Circularly Polarized (RHCP) external GPS antenna. Measuring 35 mm in length, the EtherHelix GPS is 27 percent shorter than other antennas on the market with no performance trade-offs, the company said. It is designed for high-performance, mission-critical devices such as walkie-talkies, tough books, tough tablets, first responders, public safety, military applications and more.
EtherHelix can be tuned for various satcom frequencies and various polarizations (RHCP or LHCP). EtherHelix GPS is designed using Ethertronics’ patented Isolated Magnetic Dipole (IMD) technology providing high performance and efficiency in a small form factor.
EtherHelix GPS has a high tolerance to frequency shifts given the technology’s high RF isolation, which is designed to resist antenna detuning that can otherwise impair reception. EtherHelix GPS provides exceptional coverage inside buildings, vehicles or other areas where weak signals and signal reflection occur, the company said.
“GPS capabilities are critical for first responders and military applications. It is imperative that the antennas used in these devices are high-performance, small and rugged,” said Olivier Robin, general manager Americas and Europe at Ethertronics. “EtherHelix GPS is the most recent example of Ethertronics’ leadership in developing industry-first RF solutions to provide manufacturers with a way to differentiate their products and stand out in a competitive market. Already we have seen interest in our new EtherHelix GPS antenna given its best-in-class performance coupled with its smallest occupied volume.”
Given its reduced weight and size — 11.8 g and 35 mm long by 15 mm in diameter — manufacturers benefit from simpler integration for an array of GPS devices, Ethertronics said. In addition, the new GPS antenna’s capabilities include high selectivity, which minimizes the need for additional filters. EtherHelix GPS’s ruggedized design includes IP-68 protection from dust and water, as well as a standard SMA male connector for easy integration. The RoHS Compliant antenna is designed and manufactured in the United States.
EtherHelix GPS is commercially available now and is the first in a series of antennas for mission-critical applications.
WASHINGTON, D.C. – The Federal Communications Commission today proposed rules to help emergency responders better locate wireless callers to 911. The proposed updates to the FCC’s Enhanced 911 (E911) rules respond to Americans’ increasing use of wireless phones to call 911, especially from indoors, and take advantage of technological developments that allow for more accurate location information to be transmitted with 911 calls.
The FCC’s current E911 rules require wireless providers to automatically transmit information to 911 call centers on the location of wireless 911 callers within certain parameters for accuracy. These rules, which were adopted in 1996 and underwent their last major revision in 2010, enable wireless providers to meet this accuracy standard based solely on the performance of outdoor wireless 911 calls.
However, many Americans are replacing landlines with wireless phones, and calling patterns are changing. For example, reports indicate that nearly 73 percent of 911 calls in California are made from wireless phones, and approximately 80 percent of all smartphone use occurs indoors.
In light of these trends, the FCC today proposed changes to its E911 rules to include indoor location accuracy — particularly location accuracy in challenging indoor environments such as large multi-story buildings, where first responders are often unable to determine the floor or even the building where the 911 call originated. Determining the location of indoor wireless callers is more challenging than determining an outdoor location, but innovation and technological developments in this area are making it easier to locate mobile devices wherever they are, the FCC said.
The FCC proposes in the near term that wireless providers meet interim location accuracy metrics that would be sufficient to identify the building for most indoor calls. The FCC also proposes that wireless providers deliver vertical location information that would enable first responders to identify the floor level for most calls from multi-story buildings.
In the long term, the FCC seeks to develop more granular indoor location accuracy standards that would require identification of the specific room, office, or apartment where a wireless 911 call is made, according to the statement by the FCC. These standards would rely on the advancing capabilities of indoor location technology and increasing deployment of in-building communications infrastructure.
The FCC also proposed additional steps to strengthen its existing E911 rules to ensure delivery of more timely, accurate, and actionable location information for all wireless 911 calls. In addition, the FCC is seeking comment on whether to revisit its timeframe for replacing its current handset- and network-based location accuracy standards with a single standard in light of technological developments.
While seeking comment on its proposals, the FCC also encouraged industry, the public safety community, and other stakeholders to work collaboratively to develop alternate proposals for its consideration. The FCC emphasized that its ultimate objective is that all Americans – whether they are calling from urban or rural areas, from indoors or outdoors – receive the support they need in times of emergency.
When a team of firefighters, first responders, or soldiers operates inside a building, in urban canyons, underground, in foliage, or under the forest canopy, the GPS-denied environment presents unique navigation challenges. An enhanced personal inertial navigation system (ePINS), based on a strapdown navigation solution using a mid-grade IMU and wavelet-based motion-classification algorithms, can track positions with errors of less than 2 percent of distance traveled in both indoor and outdoor environments.
By Yunqian Ma, Wayne Soehren, Wes Hawkinson, and Justin Syrstad
Numerous pedestrian navigation applications are currently available or proposed for development. Some of them include localization for coordinating firefighters, first responders, or soldiers. In these applications, the safety and efficiency of the entire team relies directly on the location and orientation of each team member. Operations in high signal interference areas such as cities, rugged terrain, forest, or indoor spaces deliver intermittent or no GPS signal. An alternative to GPS-based location is required.
In this article, we introduce an enhanced personal inertial navigation system (ePINS) solution specifically designed for environments where GPS is unavailable. ePINS combines an array of state-of-the-art sensors and fusion algorithms into a personal navigation system that provides accurate location information for pedestrian applications.
The ePINS concept.
The ePINS solution has the following benefits:
Accurate positioning in GPS-denied environments;
Small, lightweight unit can be easily carried by first responders, rescue workers, or soldiers;
Ruggedized packaging to withstand difficult first responder and military environments.
Features of the ePINS unit include:
State-of-the-art micro-electromechanical systems (MEMS) gyros and accelerometers, barometric altitude sensor, and advanced navigation software;
Advanced motion classification algorithms that accurately identify and measure user activity;
Immunity to magnetic disturbances.
Related Work
In the field of personal navigation, it is common to find systems that rely on sensors that need infrastructure (for example, Wi-Fi positioning) or sensors that actively emit electro-magnetic radiation (such as Doppler radar). These requirements are major drawbacks for communities such as dismounted soldiers in hostile environments.
Other approaches exploit the so-called Zero-velocity update (ZUPT) mechanism, which resets the inertial measurement unit (IMU) velocity errors during the stationary phase of motion. However, implementation of such schemes relies on sensors embedded in footwear, which is not readily accepted in many user communities.
To address these drawbacks, Honeywell has been developing advanced aiding techniques for personal navigation that do not rely on infrastructure and compute a self-contained, relative-navigation solution based only on passive sensors. One technique that Honeywell has developed uses displacement estimation from human-motion models. This technology has been implemented in the ePINS prototype and shows promising performance.
The human-motion model uses IMU measurements as inputs and was developed to infer distance traveled. It generates a displacement estimate that is used as a measurement in the navigation filtering process. The first version of this model was matured under the DARPA individual Precision Inertial Navigation System (iPINS) program. The iPINS system used an IMU, GPS, barometer, and motion classification to estimate a person’s position in both indoor and outdoor environments. In this system, IMU signal characteristics (e.g., peaks and valleys in the accelerations induced by walking) were exploited to differentiate between walking and running. Honeywell recently expanded the human-motion model to identify more specific motion types using a new wavelet motion classification method.
System Description
Figure 1 displays the hardware architecture of the ePINS, a small battery-powered, highly integrated electronic system. The ePINS processing platform is an ARM11-based, i.MX31 system-on-module, paired with support electronics. In addition to the processing platform, the ePINS assembly includes a MEMS IMU, a barometric pressure sensor, a digital magnetometer, and a GPS receiver.
Figure 1. ePINS hardware architecture.
The MEMS IMU provides inertial measurements for strapdown navigation. The IMU’s small package size, light weight, low power consumption, and impressive performance make it attractive for use in the ePINS system. The device is less than 5 cubic inches and weighs less than 0.35 pounds. It consumes about 3 watts of power with a typical current draw of 600mA at 5V.
The ePINS software system is shown in Figure 2. The navigation software runs within Honeywell’s Embedded Computing Toolbox and Operating System (ECTOS IIc), which provides a layered, customizable, and reusable software architecture for implementing navigation, guidance, and control software. A Honeywell-developed simulation tool for offline analysis and development of ECTOS-based software was also used in ePINS development and testing.
The ePINS demonstration device can achieve path performance of less 2 percent distance traveled for walking motion after 1 hour of operation, independent of the magnetic environment. Current performance, packaging characteristics, and interfaces are summarized in Table 1.
Table 1. ePINS performance objectives and physical specifications.
Algorithm Description
Figure 3 depicts the overall sensor integration and data processing scheme used in the ePINS device.
Figure 3. Sensor integration using the ECTOS extended Kalman filter.
Extended Kalman Filter (EKF). The EKF estimates the navigation and sensor errors and computes the resets applied to the strapdown navigation solution to increase its accuracy. Error models for the navigation sensors (IMU, barometric altimeter, magnetometer, GPS, and motion classification) are contained in the EKF. For the ePINS device, the virtual measurements from the step-length model and the strapdown navigation solution are fused by the EKF to assist in bounding the time dependent error growth of the strapdown navigator, which in turn helps maintain calibration of the inertial sensors. A key output of the EKF is the navigation confidence, which is an estimate of the accuracy of the navigation solution.
An important aspect of the EKF and step-length modeling is the residual test that the EKF supports. This test provides a reasonableness comparison between the step-length model estimate and the distance predicted by the strapdown navigation system. This capability significantly increases the robustness of the navigation solution, especially when the user is engaged in motions not recognized during motion classification.
Human-Motion Model. The human-motion model includes two components: wavelet motion classification and step-length model estimation. The wavelet motion classification identifies the type of motion the user is performing, and the step-length model acts as a virtual sensor that quantifies the motion as a distance-traveled estimate.
Wavelet Motion Classification. Human motions are very diverse and highly irregular. Determining what motion is being performed is a challenging problem of classification. Honeywell’s solution is based on wavelet transformation of IMU data. Predefined, or known, characteristics of a variety of motions (such as walking, running, crawling, etc.) are cataloged and stored to a device’s memory. Estimates of those same characteristics for a user are then computed in real time and compared to the catalog of stored information to find the best match.
Generating the catalog of stored information is an offline task that begins by “segmenting” recorded IMU time domain data into individual steps. An example of the output of the segmentation process is shown in Figure 4.
Figure 4. Segmentation of the IMU data using the y-axis accelerometer signal.
Figure 5 displays the segmentation results for two different walking styles (in red and blue) across approximately 15 example steps. As is evident from the graph, walking has characteristics that are common across users, for example, the sharp peaks in the z-axis acceleration caused by foot-ground impacts. Once the data has been segmented, a wavelet transformation on each data channel is performed. Wavelet transformation for many users over many different motion types takes place offline. Subsequently, a wavelet descriptor is built for each motion type based on the transformations into the wavelet domain. With this method, a wide variety of information (that is, descriptors) suitable for input to a classifier is captured about each motion. These descriptors are then cataloged and stored in memory on the ePINS device.
Figure 5. Sample steps for two subjects (red) and (blue).
Finally, for the online phase, the wavelet descriptor of the incoming IMU data is calculated by performing a wavelet transformation on each data channel. This descriptor is then compared to the pre-computed and stored descriptors to classify the motion. FIGURE 7 shows an example of the motion classifier output, where a running motion was used as an input. The classifier successfully determined the motion type (blue field), frequency and phase of the input motion, depicted by the tallest rectangle in the figure.
Figure 7. Classification results from a query of running at a certain frequency and phase (depicted by the dark sphere).
Step-Length Modeling. Once the current motion is identified, a step-length model specific to that motion is used to aid the navigation algorithms. The model for each motion type is obtained by first collecting data that measures step length and step frequency. From this data, the step-length models can be computed by performing a regression analysis of the step-length vs. step-frequency data. Since the step-length models act as a virtual sensor, the models must be as accurate as possible to achieve better system performance. To attain model accuracy, an accurate data collection method is needed.
For ePINS development, step-length models for multiple users have been identified from step-length and timing information using a precise GPS truth reference system. Step-length regression calculations then determine the step length as a function of step frequency (that is, inverse of the step time period). An example of GPS truth data and the corresponding regression model are shown in FIGURE 6 for walking motions.
Figure 6. Step length versus frequency for the walking of subject.
Although basic step-length models are created offline, online calibration of the step-length model can be performed by the EKF if GPS is available during operation. Online calibration tends to increase the overall position accuracy, as variations in the step-length models are likely due to slight variations in biometric differences across humans, terrain features, and even mission plans and duration.
Heading Determination. Heading initialization is one of the key concerns during system start up. In its current operational use, the ePINS device may perform a dynamic or a static initialization of heading. The static method requires the user to survey the system’s initial heading to an accuracy value that is usually specified by mission performance objectives; the absolute position accuracy is dependent upon the accuracy of the initial heading.
The dynamic method is a general method for heading initialization; it is performed without input from the user, but is possible only when GPS is available. This method of heading initialization does not use any a priori information about heading and requires an EKF implementation with a large-azimuth error model. This method requires an additional period of time in which the heading error uncertainty converges.
User Interface. During a mission, the user can interact with the navigation system and monitor its output on a display. The current ePINS prototype offers two-way communication via a serial connection. The serial communication is made wireless by the addition of a Bluetooth interface. Users can use this link to monitor the status of the navigation solution and to send commands to the device.
Honeywell has developed an application for the Android platform for this purpose. One of the key features of the interface design is that the navigation system outputs data in a standard NEMA format. Thus, publically available Android applications, not just proprietary applications, can also receive and display the navigation solution output by the ePINS device.
Honeywell’s personal navigation application displays the user’s traveled trajectory in real-time. The application can be adapted to include building floor plans as well as other navigation information.
Results
The ePINS prototype has been evaluated both in simulations and indoor/outdoor experiments. The navigation results presented here were obtained in February 2012 at a Honeywell facility (FIGURE 8). First, the user completed the heading calibration, and then online step parameter estimation in the presence of GPS was performed. Once calibration and training was completed, the GPS was disabled to simulate a GPS-denied environment outdoors. The user than transitioned to indoors (with GPS still disabled), and walked a course inside that included walking up and down stairs (FIGURE 9) and ended in a conference room (FIGURE 10).
Figure 8. Course for the Honeywell facility demonstration.Figure 9. The user walking up stairs.Figure 10. The user at the end of the demo.
Over these conditions, the ePINS system performed robustly and within performance specifications. Live demonstrations and testing showing similar levels of performance were performed at the 2012 Joint Navigation Conference (JNC) and at military test sites in California and Indiana.
Summary
The technical approach of the ePINS solution to the problem of personnel navigation in GPS-denied environments is based on a strapdown navigation solution maintained using a mid-grade IMU and advanced motion-classification algorithms. We integrated an array of sensors and software into a system that provides accurate position information and is suitable for use by first responders, soldiers, and other personnel where GPS is unavailable. ePINS works well for a variety of pedestrian motion types, including walking, running, crawling, walking upstairs, walking downstairs, sidestepping, and walking backwards. The motion classification and modeling method is extensible to other motion types.
We tested the ePINS system in indoor and outdoor environments. FIGURE 11 depicts the future ePINS concept, and TABLE 2 presents its future physical characteristics.
Figure 11. Future ePINS concept and mounting position.Table 2. Packaging characteristics of the future ePINS.
Acknowledgments
This article is based on a presentation made at ION GNSS 2012.
Manufacturers
The ePINS processing platform uses Honeywell Agile Navigation and Guidance Integrated Electronics support electronics. It includes a Honeywell HG1930 MEMS IMU, a Bosch Sensortec BMP085 barometric pressure sensor, a Honeywell HMC6343 digital magnetometer, and a NovAtel OEMStar GPS receiver.
Yunqian Ma is a principal scientist at Honeywell Aerospace. He received his Ph.D. degree in electrical engineering from the University of Minnesota, Twin Cities. He is currently the program manager of the GPS-denied navigation program and the next-generation personal navigation program.
Wayne Soehren is a senior technical manager at Honeywell Aerospace. He was the program manager for the development of Honeywell’s first MEMS-based GPS/INS, which developed the core capability now used in Honeywell’s IGS-2XX family of MEMS-based GPS/INS products. He holds an MSEE from the University of Minnesota.
Wes Hawkinson is an engineering fellow at Honeywell Aerospace. He holds a BSEE/CE from the University of Wisconsin–Madison. Justin Syrstad is a guidance and navigation scientist. He received a master’s degree in aerospace engineering from the University of Minnesota.
