Let's Talk about OrientDB Stability, Code Coverage and Bugs

After 11 months of development and QA testing, we’re thrilled to finally release Production-Ready OrientDB v2.2 GA!


We’ve worked day and night to provide our users with tremendous performance improvements. Users are reporting from 2 to 10 times the performance! We increased security so much (encryption at rest, password salt, Kerberos support, etc.) that as of date, there is no other NoSQL DBMS on the market that is more secure. (More to come on that, as a global top tier bank has chosen to partner with OrientDB as their NoSQL DBMS of choice and has helped us to improve our security even further!) Finally, we added new features we knew our users would be excited about: like the Incremental Backup, Load Balancing, Command Cache, Parallel Queries, Sequences, Reactive Live Query, Pattern Matching, Teleporter and much more.

In this period, we fixed about 2,000 issues and increased the code coverage by +11 points (from 55% to 66% of covered lines as we speak), even though the actual code base grew:


How does code coverage work?

OrientDB decided to make this test code coverage report publicly accessible to offer our users complete transparency. Want to take a look at our competitors’ code coverage to compare? I’m unable to find any that actually publish their code coverage, but please send me the links if you’re able to find that info.

Even though we already have more than 3,850 test cases, we’re constantly working to improve the quality of OrientDB by adding more and more test cases. Do you want to contribute? Help improve OrientDB’s test coverage by sending in your test cases.

Why is the average coverage so far below 100%?

These are some factors to consider that impact the average:

What about our Open Bugs?

Today we have 343 open bugs. Too many, right? A DBMS should be reliable. How can I trust a software with so many issues? Well it’s a bit more complex than that.

A young player in the DBMS space could seem like a better product due to their low number of bugs. Or a simple product that handles a few use cases can seem like they have less issues compared to one that is more complex. However, the more people that use a DBMS and the more complex the functionality and number of possible use cases, the more bugs it will have. Let’s take a look at other DBMS projects. (Click on the DBMS name below the chart to check the issues.)

*Sources: OrientDB, Cassandra, MongoDB, CouchBase, MariaDB, MySQL, Neo4j

OrientDB is way below the average. Comparing our 343 bugs with 5,252 of MySQL means we have only 6% of their amount. For Postgres, it seems that they don’t have a public issue tracker, because they handle all the bugs via email, so I’m not able to provide any number for them. (*See Update below.)

So from this analysis, is MySQL the most buggy and unstable database on the market? I’m sure you can agree that’s not the case. It’s still the most popular DBMS in the world with hundreds of thousands of users and thousands of applications and websites that are running on it.

There is no perfect software. Any complex, highly-used software like OrientDB will always have bugs due to its complexity, vast range of use cases and the thousands of users around the globe that open issues. Also, please keep in mind that OrientDB is a pure multi-model DBMS with much more functionality/capabilities than the leading document, graph and relational NoSQL DBMSs combined. When you look at it that way, our bug numbers are actually quite low. Finally, while our competitors usually take a couple of years to follow in our path in regards to features, by then we’ve already ironed out the issues in OrientDB, maintaining our lead ahead of the DBMS market and solidifying our promise to our users that OrientDB will always be cutting-edge technology.

What else is OrientDB doing to improve quality?

Are we satisfied with the amount of OrientDB bugs? Absolutely not. That’s why we continue to work hard to bring this number down to as close to zero as possible and why we are constantly improving and increasing our QA tests and documentation.

Now that OrientDB 2.2 GA is available, our Development Team has stopped development on further versions (apart from the current work packaging the OrientDB Enterprise Edition v2.2 that will be released shortly) and will spend the next month heads-down resolving issues, bugs, etc. So, if you have any problems, please send those in asap.

I’ll keep you posted about our results on this and for any questions, please don’t hesitate to write to our Open Community where more than 3,000 passionate users work together to improve OrientDB everyday.

If you have any additional questions or would like to receive more information go ahead and contact us.

*UPDATE: As of June 1st, we have 270 bugs (73 bugs fixed in 2 weeks) which is also 7 less than Cassandra.

Best Regards,

Luca Garulli
Founder & CEO
OrientDB Ltd