Ensuring production-safe DBtune
DBtune prioritizes safety in production environments through built-in guardrails. These guardrails are designed to prevent downtime and performance degradation resulting from the tuning process. Critically, users define these guardrails within the agent, enabling them to specify the conditions under which the current configuration should be flagged as unsafe. These checks are performed both during and after tuning sessions.
Guardrails during a tuning session
These guardrails are in place to prevent unexpected issues arising from new configurations being actively tested by DBtune during a tuning session.
Native memory guardrail
During a tuning session, DBtune actively monitors total RAM usage using a built-in guardrail. By default, if a PostgreSQL configuration causes memory usage to exceed 90% of available RAM, DBtune automatically adjusts to avoid that configuration. This adjustment is a seamless part of the tuning process and requires no user intervention.
Users can customize this 90% memory threshold to match their specific safety requirements. To change the memory threshold. To change the memory threshold add the following configuration in the dbtune.yml agent config file:
tuning_settings:
memory_threshold: 90
Or you can add the following environment variable:
DBT_MEMORY_THRESHOLD=90
When this memory usage guardrail is triggered (reaching the defining threshold), the agent immediately reverts to the default (baseline) configuration. To allow the system to stabilize and prevent production disruptions, the agent then waits for the duration of one "tuning iteration" before the backend suggests a new server configuration to test.
Native performance degradation guardrail
To ensure optimal performance and stability, DBtune includes a built-in guardrail that actively monitors the performance of each PostgreSQL server configuration being tested on a customer instance during the tuning session.
This guardrail triggers when:
- The Average Query Runtime (AQR) for a configuration being tested by DBtune during tuning session doubles (e.g. increasing from 50ms to 100ms).
- The Transactions per Second (TPS) for configurations being tested by DBtune during a tuning session drops below 95% of the established baseline performance.
When this performance guardrail is triggered, DBtune will proceed by testing a new recommended configuration. Unlike the memory usage guardrail, which caused a reversion to the baseline configuration, this guardrail simply prompts DBtune to explore a different tuning path.
User-defined guardrails
DBtune allows users to define their own guardrails.
Extend DBtune's protection by adding your own custom guardrails. This is useful for monitoring metrics like high disk growth rate or any other anomaly. You can achieve this by either adapting the PostgreSQL adapter or writing your own adapter.
This feature is a work in progress. More information and practical guidance on their creation will be shared as it becomes available.
Guardrails after a tuning session
Even after a tuning session ends, the DBtune agent remains active, continuously monitoring the database instance. Its purpose is to proactively alert users to any unhealthy server behavior that might emerge at any point in the future, even if it wasn’t apparent during the tuning process.
Native memory guardrail notification
Following the completion of a tuning session, DBtune continues to monitor total RAM usage with a dedicated guardrail. Unlike the active memory management during tuning, this post-tuning guardrail operates by notifying the user of any breaches in the defined threshold, without automatically altering server parameters. This aligns with DBtune’s core principle of only modifying the PostgreSQL instance during active tuning sessions. As with the previous active tuning guardrail, this notification threshold is also adjustable. As soon as a guardrail threshold is reached, DBtune takes the following two actions:
- An email notification is sent to the user’s address.
- A notification is displayed to the user within the DBtune platform.