Housekeeping Best Practices

Housekeeping is a critical part of maintaining a healthy RunMyJobs instance. At the very least, is important for Operators and Administrators to frequently look for red dials in the Housekeeping Dashboard, so that you can address housekeeping issues as they arise. It's also important to configure your retention settings to warn you if you start running out of space.

However, a smarter way to handle housekeeping issues is to prevent them. The topics in this section describe how to tune your RunMyJobs instance to achieve better performance by preventing RunMyJobs database tables from becoming too large in the first place.

To avoid housekeeping issues, Redwood recommends the following:

  • Consult the Housekeeping Dashboard frequently.

  • Use the System_Defaults_System Process Definition to set a base retention policy for your RunMyJobs data.

  • When you develop Process Definitions and Chain Definitions, always keep in mind the number of records they generate. If a Process Definition runs frequently, consider a more restrictive Retention configuration that reduces the number of processes of that type that are retained.

  • When configuring restart behavior for a Process Definition or Chain Definition, keep in mind that each process restart generates a log. If you configure a process to retry on error with no delay, you run the risk of generating a huge number of logs if that process can't complete successfully. Consider setting a Maximum Starts value to prevent a process from retrying endlessly.

Note: RunMyJobs SaaS customers always have a healthy environment, and will be warned when the Cloud Services team notices anomalies that need to be addressed.

Key Considerations

There is more to housekeeping and archiving than just deciding how long you want to keep your data. The following considerations should be taken into account when looking at housekeeping and archiving requirements.

  • Why: Why do you want to keep and archive data? Make sure you understand your company policy. You might not be required (or even allowed) to retain data for a specific period.

  • What: RunMyJobs does not store customer data, but it generates a lot of execution metadata. Do you need all of it, or do you only care about processes that fail? Keep in mind that customer data must be archived on customer systems, not from within RunMyJobs.

  • Where: Your archive location should be safe, maintained, and available. Will data be stored on-site or at a different location? Follow your company standard.

  • When: How often do you need to access archived data? Is the archive intended for regular viewing, or should it only be restored when required?

The Big Picture

The image below shows the lifecycle of execution data. From left to right:

  • Process Execution: Execution data is stored in RunMyJobs.
  • Data Available in Redwood (Cloud): The execution data is available in the cloud. RunMyJobs stays clean, fast and stable by following retention rules (keep clauses). Any data that must be kept longer can be archived in a customer's archiving system before a data purge (for example, using Redwood Reports).
  • Data Purge: Redwood's standard housekeeping cleans up the execution data based on the defined retention.
  • Data Available in Customer's Archiving System: You will have the data in your archiving system available for long-term auditing.
  • Archive deletion: Even archiving systems will, in most cases, have retention and must be periodically cleaned up. This task can also be automated.