Benchmarks
YCSB General Purpose and Internal Benchmarking
Technical deep dive to optimize the most demanding workloads.
YCSB General Purpose Benchmark
All numbers for competitors originate from benchANT's database rankings. Source code for RogueDB's benchmarks can be found here.
Overview
YCSB (Yahoo! Cloud Serving Benchmark) is an open source tool used to evaluate the performance of databases under different simulated workloads. The GitHub repository can be found here. There are a vast number of configuration options with the option for database providers and the general public to adapt the framework to a specific database. For the purpose of RogueDB's benchmarking, we utilize benchANT's configuration settings as the specification for our internal implementation given their publication of numbers for various competitors.
Those workload settings are the following:
- 50 workers (eg. 50 threads) sending requests to the database.
- Read and write ratio is split 50:50.
- Read access matches a Zipfian distribution with alpha equal to .9.
- Insertions are sequentially ordered.
- Only 1 message type (eg. data schema) is used throughout.
- Data comprises of 500 bytes (10 string fields of 50 bytes) and a primary key.
- Initial database is populated with 2.5GB of data (eg. 5 million objects).
- Data redundancy is local only (eg. no regional data replication).
- Standard storage is used (eg. differs based on cloud provider).
- Database and client are assumed to be hosted in the same region and zone.
- Databases configurations are vanilla out of the box settings (eg. no modifications).
- Client size is 8 cores (eg. 16 threads).
- Database size is 4 cores (eg. 8 threads).
- Connections use database specific drivers or JDBC.
Caveats to be addressed:
- RogueDB utilizes the latest AMD Turin CPUs.
- benchANT utilizes Intel Sapphire and Intel Cascade CPUs.
- RogueDB uses Google Cloud exclusively.
- benchANT uses AWS, Azure, and Alibaba for benchmarks.
- RogueDB utilizes C++ and an equivalent benchmark stripped of all abstractions to reduce external factors on performance.
- YCSB utilitizes Java and several levels of abstraction that may pose limiting factors to the overall performance of the benchmarked databases.
Our selection criteria for competitor's benchmarks was simply to pick the one with the highest throughput with the same number of cores (eg. disregard version, cloud provider, and edition). We did this to give competitor's the most favorable numbers possible. RogueDB strives to be fully transparent with the benchmarking of our database, If you find errors here or in the benchmark code, please let us know at contact@roguedb.com. We will promptly address anything the team may have missed when putting together this data.
Performance Comparisons
RogueDB achieves 452,039 op/s in the benchmark. Total performance increase ranges from 8.5x to 56.5x increase from the nearest competitor to the farthest competitor. The objective of the general purpose benchmark is to demonstrate overall database performance for typical workloads. In addition to measuring throughput, these benchmarks often measure latency. Our benchmarks do not currently measure latency, but we plan to measure this metric in the future.
For reference, competitors' throughput from benchANT:
- Couchbase: 53,050 op/s
- PostgreSQL: 40,032 op/s
- MongoDB: 21,849 op/s
- Cassandra: 21,995 op/s
- ScyllaDB: 17,529 op/s
- MySQL: 18,906 op/s
- CockroachDB: 13,364 op/s
- CrateDB: 7,995 op/s
Modified YCSB
Efficient network transmission of data poses a significant bottleneck on total throughput of RogueDB. We have written several articles on the subject in our blogs and encourage reading for a technical deep dive. The high level summary is batching increases total throughput by an order of magnitude both for gRPC and RogueDB.
To demonstrate the effects, we modified the YCSB General Purpose benchmark to measure throughput for payload sizes of 1, 10, 100, and 1,000. The benchmark also isolates Read and Write into standalone measurements to highlight the effects on each operation. Combined throughput for any ratio can be expected to fall somewhere between the individual throughput based on bias towards either Read or Write.
Throughput as measured:
- Read Batch 1: 452,039 op/s
- Write Batch 1: 430,425 op/s
- Read Batch 10: 1,129,685 op/s
- Write Batch 10: 753,601 op/s
- Read Batch 100: 1,170,998 op/s
- Write Batch 100: 885,505 op/s
- Read Batch 1,000: 1,117,608 op/s
- Write Batch 1,000: 914,752 op/s
Single message load to batch size of 100 for Read operations provides around a 2.5x throughput increase. For Write operations, single message to batch size of 1,000 provides around a 2.1x throughput increase. This benchmark demonstrates the upper threshold of operations per second for RogueDB is much higher than the normal YCSB General Purpose benchmark initially indicates.
Pre-existing benchmark data of a workload similar to this one does not appear to be available for competitor databases. The open source YCSB library exposes workload settings that appear to allow modification of message load size. However, the purpose of this benchmark is not intended to compare against competitors but rather give an indication of the upper limit on total throughput for RogueDB.