Mark BurgessWe built the OCI plugin for Checkmk because we run Checkmk across every environment we manage and did not want a separate monitoring stack for the Oracle Cloud part of it. It has run against our own managed-service tenancies for some months and has been open source, GPL-2.0, on GitHub and the Checkmk Exchange since July. What follows is what we learned building and running it, written for someone about to point Checkmk at an OCI tenancy who wants to get the structure right first time.
The OCI Monitoring documentation explains the metric namespaces, the alarm model and how to query metrics. The Checkmk documentation explains how a special agent, the check API and rulesets fit together. Both do their job. This post covers a different question: which signals from an OCI tenancy matter operationally, and how to lay the monitoring out so that what lands in Checkmk is useful rather than merely numerous. The plugin is the worked example; most of the decisions apply whether you use it or build your own.
Here’s what matters.
Decide What a Control-Plane View Is For
The first decision is scope, because it shapes everything else.
OCI Monitoring collects performance metrics from services and raises alarms on them. The plugin does something different. It reads the control plane: lifecycle state, configuration, backup and replica posture, load balancer health, the tenancy’s cost run-rate, and what Oracle is telling you about your regions. It issues list_*, get_* and request_summarized_usages calls and nothing else – never the Monitoring metrics API, and never anything that changes state.
That is a deliberate boundary, not a gap. Performance metrics belong with the service that produces them; what a Checkmk operator is usually missing is the other half. A stopped instance that should be running, a volume whose last backup is two days old, a cross-region replica that has quietly stopped syncing, an Autonomous Database in AVAILABLE_NEEDS_ATTENTION, a compartment whose spend has crossed the line you set – none of those are performance metrics, and none appear in the console unless someone goes looking.
If you already run the Checkmk agent on your OCI compute hosts, or the Checkmk Oracle plugin on Base Database Systems, keep doing so. The plugin adds the tenancy-level view around those hosts; it does not replace what runs inside them.
Treat OCI Monitoring as the metrics layer and Checkmk as the operational layer. Do not try to make one do the other’s job.
What the Plugin Covers
The plugin is a Checkmk special agent. Nothing is installed inside the tenancy; the site authenticates with a read-only API key and the checks run on the Checkmk server. It works on Checkmk 2.3 and later on every edition including Raw, and has been tested against 2.4. The oci Python SDK is not bundled with Checkmk and has to be installed into the site’s Python.
Each service family is a separate toggle in the rule:
- Compute – lifecycle state, shape, OCPUs and memory, attached block and boot volumes, cost per instance
- Block and boot volumes – size, VPUs, latest full and incremental backup timestamps, cross-region replica state and sync gap
- Volume groups – group state, backup policy assignment, coordinated backup age, replica state and sync gap
- File Storage – state, size, metered usage and cost, replica sync
- Object Storage – bucket size, object count, replication policy, storage tier (metadata only; object contents are never read)
- Load balancers – overall health and per-backend-set health, naming the backend sets in warning or critical
- Oracle Database – Base DB Systems (state, cores, storage, licence model, cost) and their databases (auto-backup configuration, recovery window, last backup duration, management status)
- Autonomous Database Serverless – ATP, ADW, APEX and AJD; state, compute and storage, auto-scaling, backup retention, Data Guard, mTLS, cost
- Virtual Cloud Networks – VCN and subnet inventory with security-list resolution
- Tenancy usage – month-to-date cost for the tenancy and per service, with thresholds
- Announcements and region status – the OCI Announcements service and the public region status page, each as a Checkmk service
- Topology – network and resource datasets for Checkmk’s built-in topology viewer
Resources Become Piggyback Hosts
Compute instances, databases, load balancers, file systems and volume groups are emitted as piggyback hosts named after the resource, so an instance called admin-cmk-01 becomes a Checkmk host of that name. If the Checkmk agent already runs on that instance under the same name, the OCI services simply join the OS services on the existing host. Autonomous Databases are named <db_name>-<region> because the same database name can exist in more than one region.
Each piggyback host also carries host labels: oci_service_type on everything, plus workload and database-name labels on Autonomous Databases. This is the part we use most outside monitoring. Checkmk is the dynamic inventory for our Ansible runs against OCI, with fleet jobs targeting hosts by label straight off the monitoring data. The plugin stays read-only; Ansible does the acting, and that separation is what keeps the monitoring credential at inspect-and-read.
Aggregates Go to Inventory, Not Metrics
Per-region and per-compartment totals – instance counts, running versus stopped, OCPUs, storage, cost – are written to the HW/SW inventory tree under OCI/aggregates/ rather than emitted as metrics. An earlier version produced a metric per region-compartment-service combination, which meant RRD files multiplying faster than anyone would look at them. Inventory gives you the same data for reports without the storage cost.
If a service family is not in the list above, the plugin does not monitor it.
Authentication and Least Privilege
The plugin authenticates as an OCI user with an API signing key. That is the only method it supports at present, so plan the identity side accordingly.
Create a dedicated user in a dedicated group, with no console password, so the credential can be rotated or revoked without disturbing anyone’s console access and the audit trail contains nothing but monitoring calls. Attach the policy to the group at the tenancy root, not to the user.
Three policy shapes work, and the choice is a compliance decision rather than a technical one.
The broad policy is three statements: inspect all-resources, read all-resources and read usage-report, all in tenancy. This is what most deployments should use and what the rule assumes.
The least-privilege policy grants read on exactly the service families the plugin touches (instance-family, volume-family, database-family and so on), plus inspect and read on compartments and inspect on tenancies. Any block whose service toggle you leave off can be dropped; the toggle-to-statement mapping is in the plugin’s IAM documentation.
The compartment-scoped policy restricts read all-resources to named compartments. Two statements stay tenancy-wide: read usage-report, because the Usage API does not accept compartment scoping, and inspect compartments, because the plugin cannot walk a tree it is not allowed to enumerate.
The manage verb is never required. If a policy you have been handed contains it, the policy is wrong.
The user OCID, tenancy OCID and key fingerprint go through the Checkmk password store, so they do not appear in process listings. The private key is a file on disk – the password store cannot hold a multi-line PEM – under the site user’s home, mode 600, distributed by whatever you already use for secrets. Treat it as an SSH private key.
For firewall allow-listing, the agent talks HTTPS on 443 to the per-service *.oraclecloud.com endpoints (identity, iaas, filestorage, objectstorage, loadbalancer, database, announcements) and to usageapi in the home region only. The region status check reads the public status page and needs no IAM at all.
Run the agent by hand as the site user before you touch discovery. Each failure has a signature. 401 NotAuthenticated: the private key does not match the public key on the user, or the fingerprint is wrong; regenerate the fingerprint locally and compare. 404 NotAuthorizedOrNotFound: the policy is not attached, or is attached at the wrong level; check it is on the group at the tenancy root. 429 TooManyRequests: a rate limit, covered below. The log is var/log/agent_oci/special_agent_oci.log under the site.
Fix authentication on the command line. Discovery will not tell you anything the log has not already told you.
Compartments, Regions and Discovery
By default the plugin monitors every compartment under the tenancy root in every subscribed region, which is usually what you want. When it is not, the filter semantics matter, because getting them wrong does not fail – it silently monitors less than you think.
The compartment filter is a comma-separated list of compartment OCIDs or names, matched exactly and case-sensitively against both. A name used on more than one compartment matches all of them, which is useful for fan-out and a trap otherwise; use the OCID to disambiguate. Sub-compartments are not included automatically. If you want Production and its children, list each child.
The diagnostic signature of a wrong filter is a warning in the agent log – Compartment filter entry matched no compartment – and a host with fewer services than you expected. Nothing turns red. Check the log the first time you apply a filter, every time.
Regions work the same way. Leave the region list empty and the plugin discovers subscribed regions from the tenancy; specify a list and any entry that is not subscribed is skipped with a warning. The home region is always required, because that is where the Usage API lives.
Regions are processed in parallel, each worker writing to its own buffer, with the buffers written out in stable region order once every worker has finished. Letting threads write to stdout as they go interleaves piggyback headers and produces output Checkmk parses into nonsense. The rule exposes the worker count, default four.
Discovery itself is ordinary Checkmk. Assign the rule to a host representing the tenancy, activate, run service discovery. Tenancy-level services land on that host; resource-level services land on the piggyback hosts. In a distributed setup the oci SDK has to be installed on every site that executes the agent, not just the central one; that is the usual reason a remote site’s OCI host shows nothing.
The Cost of Polling
An OCI tenancy is not a server. Every check cycle is a set of API calls against a metered, rate-limited, sometimes slow control plane, and most of the plugin’s design is about keeping that honest.
Cache by How Often the Data Actually Changes
The plugin uses tiered cache TTLs based on how often the underlying data can change. Resource lists refresh at the default five minutes, backup information at twenty, per-resource cost hourly. Identity-layer data – compartments, tenancy metadata, region subscriptions – and backup policy assignments refresh once a day, because they change on the order of days or never. Tenancy-wide usage also refreshes daily, because the Usage API’s own data updates roughly once a day and polling faster returns the same numbers.
The failure you are avoiding is an agent that hits the Identity API for the full compartment tree on every run – on a large tenancy, the difference between an agent that completes and one that times out.
Fetch Cost in Bulk
Cost attribution is the expensive part. One Usage API call per resource is a few hundred calls on a modest tenancy against an API with a low quota. The plugin instead makes one tenancy-wide call grouped by resourceId, indexes it by OCID, caches it for an hour, and looks each resource up in the index.
The catch is that a group_by resourceId query omits resources with no billable cost – free tier, just created, or merged into another dimension. For those the plugin falls back to a per-resource query and caches the answer individually. A resource reporting no cost is usually a genuine zero, not a lookup failure.
Bound the Retries
The OCI SDK’s default retry strategy is generous. When a service returns HTTP 500, exponential backoff can consume more than a minute on a single call, and an agent making many calls will blow through Checkmk’s agent timeout long before OCI recovers. The plugin overrides this with two attempts and decorrelated jitter between one and ten seconds, retrying on 429 and any 5xx, with ten seconds to connect and thirty to read.
The OCI APIs are rate limited – the Usage API in particular is metered per tenancy in the home region, and the plugin is built to touch it rarely: the agent runs every five minutes, but tenancy and per-service usage are fetched once a day and the bulk per-resource cost index once an hour, with everything in between served from cache. If you see persistent 429s in the log, reduce region parallelism first, then check whether something else is also polling usage.
Paginate Everything
Every list_* call goes through the SDK’s pagination helper. The default per-call limit on most OCI list endpoints is around a hundred items, and an unpaginated call returns the first page with no error. The signature is a compartment where resources beyond the first hundred never appear in discovery.
Run Once, Not Twice
A full fetch over two regions took roughly forty seconds in our environment. Checkmk’s automation helper runs a small number of workers, so a GUI discovery preview racing the scheduled check can tie up every helper on duplicate OCI traffic and produce “Could not connect to automation helper” errors that look like a Checkmk fault. The plugin serialises identical invocations with a lock: a second caller for the same scope gets the previous run’s cached output.
Poll less than you think you need to. The data does not change as fast as a server’s does, and the API is not free.
Thresholds: Start From the Service’s Own Failure Modes
The plugin ships with default thresholds, and they are placeholders. What matters is the model behind them, the same across every check: map lifecycle state to Checkmk state according to what the OCI service itself considers healthy, treat configuration as something you assert rather than observe, and use time-since as the signal for anything that is supposed to recur.
Lifecycle State Is the First Result
Every resource check yields lifecycle state first, and the mapping is service-specific. An Autonomous Database in AVAILABLE or STOPPED is OK; FAILED, TERMINATED, INACCESSIBLE and AVAILABLE_NEEDS_ATTENTION are CRIT; transitional states – PROVISIONING, SCALING, UPDATING – are WARN so they are visible without paging anyone. A stopped ADB is OK because stopping it is a normal cost-saving action and the check cannot know whether it should be running.
Compute instances take the opposite approach: the rule carries an expected lifecycle state, default RUNNING, and any other state is CRIT, overridable per host for instances meant to be off.
The signature of getting this wrong is either a wall of CRIT from resources behaving as designed, or a resource that has sat failed for a week while the check reported OK. Read the service’s lifecycle states before you decide which ones are bad.
Configuration Is Asserted, Not Measured
Several checks compare what OCI reports to what the rule says should be true: licence model on a DB system, workload type, Data Guard and mTLS on an ADB, backup deletion policy and full-backup day on a database, whether auto-scaling is enabled. A mismatch is WARN. These are drift detection: someone changes a setting in the console, the rule still says what the standard is, and the service stays yellow until one of them is corrected.
Set assertions for what your platform standard actually specifies and leave the rest at “no validation”. An assertion you have not thought about is a WARN nobody will act on.
Backups and Replicas Are Measured by Age
The shipped defaults reflect the cadence of the underlying mechanism. Volume group backups warn at 25 hours since the last backup and go critical at 27, assuming a daily policy. Volume group replicas warn at 5 minutes since last sync and go critical at 10. Block volume and File Storage replicas warn at 90 minutes and go critical at 120, because the sync cadence is coarser.
If your backup policy is weekly, the 25-hour threshold will page you every day and you will learn to ignore it, which is worse than no check. Set the gap to your policy plus a margin for the job to complete. A replica not in AVAILABLE state is WARN regardless of sync age; a replica that is provisioning or faulty is not one you can fail over to.
Cost Thresholds Are Governance, Not Alerting
Per-resource, per-service and tenancy cost thresholds are month-to-date figures, and the shipped defaults are arbitrary. Set them at the level where you would want to know – for most teams a service or compartment budget, not an individual instance. A threshold crossing mid-month tells you the run-rate is wrong while there is still time to act.
Autonomous Database and What Actually Matters
If you are running Autonomous Database Serverless, the plugin is the only view Checkmk can have of it. Monitoring is API based because there is no host to install a Checkmk agent on to run the Oracle plugin, so the control plane is the whole picture, and it is worth being precise about which of the many fields the API returns matter.
State comes first, with the mapping above. AVAILABLE_NEEDS_ATTENTION is the one to take seriously: the database is up but something requires action, and it is easy to miss in a console showing green.
Storage is reported as allocated terabytes and as used percentage, the latter against 80 and 90 by default. Alert on the percentage: with auto-scaling it tells you the growth rate; without it, it is your only warning before the database stops accepting writes.
Backup retention is checked with lower thresholds: warn below 30 days, critical below 7. It is the one place in the plugin where a lower number is worse, because retention is a setting a well-meaning person reduces to save on storage without appreciating what it removes from the recovery window.
The assertion checks are where the value is for anyone running ADB to a standard. Expected workload type catches a database provisioned as ADW when the standard says ATP. Expected Data Guard state catches a production database whose standby has been disabled. Expected mTLS catches a database opened to one-way TLS to make a client work. Each is one rule setting, WARN on mismatch, and none is something the console will tell you unless you open the database and read the page.
For Base DB Systems the picture is split. The OCI plugin reports the system and the database’s OCI-managed backup configuration – auto-backup enabled, deletion policy, full-backup day, recovery window, last backup duration. The Checkmk Oracle plugin on the DB host reports everything inside the database. Run both; neither is complete on its own.
Monitor ADB for state, storage percentage, retention and drift. Everything else is inventory.
The Bottom Line
OCI already has a metrics service. What it does not have is a place where the state, configuration, backup posture and cost of a whole tenancy sit alongside everything else you monitor, with thresholds you own. That is the gap the plugin fills, and every design decision in it – read-only credentials, tiered caching, bulk cost lookup, bounded retries, labelled piggyback hosts – exists to make that view cheap enough to run every five minutes and accurate enough to act on.
Get the identity right first and verify it from the command line. Set thresholds from the service’s own failure modes, not from the defaults. Poll the control plane like the metered API it is.
Source and documentation are at github.com/burgesssystems/agent_oci; the MKP is on the Checkmk Exchange.
Need help monitoring OCI with Checkmk?
We built the plugin and we run it in production. If you want it deployed and tuned against your tenancy, integrated with your alerting, or you need someone to run the Oracle Cloud platform itself, we offer a 30-minute call – we will look at your environment and tell you plainly what it needs.
