What you’ll learn:
- What are the database management system (DBMS) requirements for complex real-time sensor data fusion?
- What application modifications are needed if a DBMS isn’t being used?
- Different definitions of real-time.
Time-critical sensor data fusion has a range of use cases. Readers are already familiar with many of them: any autonomous vehicle (land, sea, air, undersea, space,) power generation and distribution, battlefield management, and many others.
In simple terms, these systems acquire data from multiple types of sensors and other sources, then conduct some form of analysis on the data to determine some action. For this to work, the system needs to be looking at a coherent set of data from a consistent time slice. In other words, it would be a bad thing if an advanced driver-assistance system (ADAS) system was looking at LiDAR data from one time slice and the camera or GPS data from another time slice.
Time-critical sensor data fusion requires an approach to data management that incorporates awareness of real-time constraints. This can be handled entirely within the application, but that necessitates recreating all or most of what database systems already do for you. However, if a conventional commercial database is used for such systems, it’s up to the developers to “bolt-on” the temporal consistency of the data; a conventional DBMS has no notion of real-time time constraints.
Real-Time Constraints for a Database
A better approach is to build awareness of real-time constraints in a database system. Some of the considerations are:
- Database transactions must have a priority.
- Database transactions need to have a deadline.
- Database transaction schedulers must consider the priority and deadline of transactions and schedule transactions to maximize the number of transactions that meet their deadline. Conventional database systems merely accept transactions in the order in which they occur.
- Instrumentation should be added to the database system so that it can monitor a transaction’s progress toward the deadline.
Suppose the database runtime determines that a transaction will not complete before its deadline. In that case, it must terminate the transaction early enough to guarantee that rolling back the transaction will be completed within the deadline. In real-time systems, transactions can never be “late” (exceed their deadline). They must either commit or roll back before the deadline.
Illustrating Real-Time Transactions and Database Use
Illustrating the transaction timelines (see figure) for conventional and real-time databases in several scenarios is as follows:
In the first scenario (A), a conventional database uses a pessimistic concurrency model. All tasks competing for the same resource (e.g., a database or table lock) will proceed sequentially in FIFO order.
In the second scenario (B), a conventional database uses an optimistic concurrency model where the first and second tasks compete for the same resource (e.g., updating the same row of a table). All tasks will proceed in parallel, but one of the first two tasks will be notified of a conflict and must retry its transaction.
The third case illustrates what happens inside a time-cognizant (real-time) transaction manager when three tasks start transactions at about the same time (C): All three transactions (T1P2, T2P1, and T3P2) get put into the queue.
Then (D), the transaction manager schedules the transactions according to their priority and deadline.
Because T1P2 was started first and no other transactions were running at that moment, it began to execute immediately. But T2P1 entered the queue immediately after. The scheduler recognizes that T2P1 has a higher priority than T1P2, and the time to rollback is less than T2P1’s deadline, so T1P2 is preempted, and T2P1 executes. T3P2 started last and has a lower priority than T2P1, so it waits in the queue regardless of its deadline. After T2P1 is complete, T3P2 is scheduled because it has a closer deadline than T1P2.
In the absence of a real-time database system, these things must be addressed at the application level. It takes time to design and implement it, and that’s time spent on tasks not directly related to the application’s responsibility.
Doing this requires additional resources (and the associated cost) or takes time away from implementing the core functionality of the target system (e.g., driving the car). In turn, it protracts the development schedule, creating a difficult-to-measure but still authentic opportunity cost. Lastly, we know from years and years of studies that defects (bugs) are costly, and the later in the development cycles that they’re discovered, they become ever-more costly.
The Need for a "Real" Real-Time Database
Developers of mission-critical real-time systems should consider a real-time database system that has already undergone extensive unit testing, integration testing, and robust quality assurance. It thus avoids the costs of remediating defects throughout the development cycle introduced through an in-house development of the requirements outlined above.
Developers should take care not to confuse marketing with capability. The term “real-time” is used by many database system vendors to promote solutions that aren’t actually real-time. In this context, the term refers to “live” data or analytics. And “real-time” means without the delay of storing the data and then re-reading the data before conducting analytics, or updating a query’s result set “in real-time” without the need to re-query the database system.