Don’t Get Bogged Down in 3rd Party RTOS Code!
January 09, 2024
Blog
It’s midnight the day before the product release and your system just crashed. As you follow the execution back from the point of the fault, you find yourself deep inside the Real-Time Operating System (RTOS) code. If you don’t have professional support, your only option is to try and figure out why the RTOS is failing yourself. Of course, you have to first figure out how the RTOS works to even have a chance at understanding what is going wrong. If you are fortunate enough to figure this all out, you are then tasked with trying to fix the issue – and then ensure the fix is propagated back into the next 3rd party RTOS release.
Fixing bugs in unrelated code is expensive
Fixing an issue in the RTOS takes considerable time, and time is quite expensive. The median, fully-loaded embedded software engineer salary last year was roughly $181,269 (based on Glassdoor). If the RTOS bug takes you two days to fix, your direct cost is roughly $1.5K. Of course, things get much worse if the whole project is blocked.
For a typical development team of eight software engineers, each day the project is blocked costs on the order of $6K (not including indirect costs associated with schedule delays). Also remember that these costs are just for one bug with a fairly rapid two-day resolution. Things don’t always go so smoothly in the real world!
How many bugs are there?
We developers know there are bugs lurking in every piece of software. No exceptions. That said, the exact number of defects varies greatly on the quality and maturity of the specific RTOS. According to the 2014 Coverity Scan Open-Source Report, the defect density of open-source software was roughly .6 defects per 1,000 lines of code (LOC), which means there is more than 1 bug for every 2,000 LOC.
Most RTOS distributions contain many thousands of lines of code. Given that, it is likely there are bugs in the RTOS you are using. It’s also worth mentioning that this defect density number is in the context of static analysis and therefore doesn’t account for timing and other complex run-time interaction classes of bugs, which are most prevalent in RTOS software.
What can be done?
We know that there are bugs in all software and fixing them is expensive. What can we do? Investing in professional RTOS support (for both commercial and open-source software), offers many valuable benefits:
Expert Assistance:
- Professional RTOS support provides access to experienced professionals who specialize in RTOS software. Their expertise can be crucial in resolving complex issues, optimizing configurations, and providing guidance on best practices. In some cases, the guidance alone prevents costly problems down the road.
Timely Issue Resolution:
- Most professional support packages come with a Service Level Agreement (SLA) that guarantees response times for issue resolution. This ensures that if you encounter problems or have queries, you'll receive prompt assistance, minimizing downtime and potential disruptions. Remember the average $6K per-day cost if the project is stalled on an RTOS issue – so timely response is all important.
Tailored Solutions:
- Support services can provide customized solutions and recommendations tailored to your specific needs and use cases. They can assist in optimizing configurations, implementing features, or resolving issues specific to your environment.
Updates and Security:
- Many professional RTOS support services offer timely updates, patches, CVE notifications, and security fixes – even for open-source RTOS software. This ensures that you're using the latest stable versions and are protected against security vulnerabilities.
Legal Compliance:
- Some open-source licenses come with specific obligations or restrictions. Professional support can help ensure that your use of the software complies with licensing terms, minimizing legal risks.
Training and Resources:
- Support services often have options for training programs, supplemental documentation, and resources to help your team understand and utilize the software more effectively. This can improve your team's efficiency and reduce the learning curve and help improve your product’s quality as well as your time-to-market.
Stability and Reliability:
- By having a dedicated support team behind the software, you gain confidence in its stability and reliability. Knowing that you have experts to rely on can provide peace of mind when using critical open-source tools. Professional support acts like a development insurance policy.
Professional support saves money and protects your schedule
The cost of most professional support packages is often less than the cost of having your development team stalled for just one day. Professional support also helps your team manage the RTOS source code integration, including notification of new updates, patches and CVEs. There are also indirect financial benefits in helping your team create a better-quality product with reduced time-to-market. In most situations, purchasing professional RTOS support saves money!
Bill has been in the commercial RTOS space for over 30 years - first with Accelerated Technology (acquired by Siemens) and then with Express Logic (acquired by Microsoft). Bill was the sole author of Nucleus and ThreadX. Bill’s latest endeavor is PX5, where you can find his latest creation – the PX5 RTOS!
