Big data and the IoT have emerged because for the first time, you are able to move computing power from massive servers to really cheap machines. Some data systems can have hundreds of thousands of end points, moving petabytes of data in real time.

This new frontier of information offers new opportunities to those who can meet the new challenges your database must be able to manage. Here are three big challenges for Big Data and three vital components any IoT Data System must have.

What Was Science Fiction Yesterday is Now Reality

FitBit is a great example of an IoT data hierarchy. A simple device attached to your wrist can measure your heart rate in real time. For our example, let’s assume this edge point can take a measurement every 30 seconds. It will transmit that information to an application on your Smartphone, which can store the data, process it, even send it to your file in your doctor’s systems.

A Smartphone is a full-blown computer, having 100,000 times the computing power of the systems that put a man on the moon. This gives you unprecedented amounts of computing power at the edge. You can take years of hourly data and create an algorithm that will anticipate when your heart rate is likely to go up, and your Smartphone application will alert you when to drink some green tea, go for a brisk walk, or take your medicine.

Three Challenges to an Effective IoT Data Hierarchy

Connectivity. What happens when I drive through a tunnel and my Fitbit is disconnected? What if I go hiking for a week and my device records the data, but is unable to transmit it? A database must be able to work offline and communicate all the data it collected in order to properly update the entire system when connection to the network is restored.

Unexpected Updates. What happens when data isn’t transmitted for a long time? What happens when decisions are made based on tremendous amounts of information, but still incomplete?

For example, I have a month of data on my heart rate that is stored in my Fitbit device. My wife gets me a new model and I put the old one, with this data, in a drawer. Three months later it breaks, and I start wearing the old one again. The moment I put it on, it sends this data, now 3 months old, to my doctor’s application.

This is where it gets hairy.

What happens if I had a doctor’s appointment while my first Fitbit was sitting in the drawer with last month’s data. What if the data available to him says, prescribe 100mg of this medicine? Then, at my next appointment 3 months later, after all the data is updated, the retroactive updates tell the doctor that he should have prescribed me 200mg?

He needs to be able to prove that he made the right decision with what he had available at the time he made it. A database must be able to tell you the state of your data at any point in time for auditors, regulators, even opposing counsel.

Devices that age in dog years. Edge processing almost guarantees you will eventually process information on older devices. Say you are tracking sales based on information coming in from over 50,000 cash registers throughout your global operations. How often will you replace all of those cash registers? Maybe once every decade. That means edge devices will get old often and you need a database that can work on hardware that will inevitably become antiquated.

Conclusion

Processing Big Data gives you a new level of precision in diagnosing problems and developing better targeted solutions. The challenge with this new technology is to deploy the right software that is best suited to it.

Oren Eini, CEO and founder of RavenDB, has more than 20 years of experience in the development world with a strong focus on the Microsoft and .NET ecosystem. Recognized as one of Microsoft's Most Valuable Professionals since 2007, Oren is also the author of "Inside RavenDB." An avid blogger, you can also find him under his pseudonym as Ayende Rahien at http://www.ayende.com/Blog/.