Not long ago we came across an issue with database scalability when working on a project for a client with expertise in data-driven management solutions. It threw both business growth and the project’s future into question. With a total size of around 3TB and number of rows that strive to 3 billions, we reached the limit of what a single-node Postgres database instance could do (in an acceptable manner). Read replicas were not effective because the application is write-intensive, and a significant lag was in evidence between the main node and replicas.
We tried various optimization techniques including ones from Postgres and in-memory processing and calculations, but we were still pushing the boundaries of scalability. As shown in the graph below, CPU was hit hard several times during the month.
The average number of disk read operations per second shows that out working set was almost completely in memory on a daily basis. Even we tried with scaling up the DB instance class to a cla…
We tried various optimization techniques including ones from Postgres and in-memory processing and calculations, but we were still pushing the boundaries of scalability. As shown in the graph below, CPU was hit hard several times during the month.
The average number of disk read operations per second shows that out working set was almost completely in memory on a daily basis. Even we tried with scaling up the DB instance class to a cla…