Housekeeping Best Practices

Housekeeping is a critical part of maintaining a healthy RunMyJobs instance. At the very least, is important for Operators and Administrators to frequently look for red dials in the Housekeeping Dashboard screen (Configure > Control > Housekeeping Dashboard), so that you can address housekeeping issues as they arise. It's also important to configure your retention settings to warn you if you start running out of space.

However, a smarter way to handle housekeeping issues is to prevent them. The topics in this section describe how to tune your RunMyJobs instance to achieve better performance by preventing RunMyJobs database tables from becoming too large in the first place.

To avoid housekeeping issues, Redwood recommends the following:

  • Consult the Housekeeping Dashboard screen frequently.

  • Use the System_Defaults_System Job Definition to set a base retention policy for your RunMyJobs data.

  • When you develop Job Definitions and Workflow Definitions, always keep in mind the number of records they generate. If a Job Definition runs frequently, consider a more restrictive Retention configuration that reduces the number of Jobs of that type that are retained.

  • When configuring restart behavior for a Job Definition or Workflow Definition, keep in mind that each restart of such a Definition generates a log. If you configure a Job to retry on error with no delay, you run the risk of generating a huge number of logs if that Job can't complete successfully. Consider setting a Maximum Starts value to prevent a Job from retrying endlessly.

Note: RunMyJobs SaaS customers always have a healthy environment, and will be warned when the Cloud Services team notices anomalies that need to be addressed.

Key Considerations

There is more to housekeeping and archiving than just deciding how long you want to keep your data. The following considerations should be taken into account when looking at housekeeping and archiving requirements.

  • Why: Why do you want to keep and archive data? Make sure you understand your company policy. You might not be required (or even allowed) to retain data for a specific period.

  • What: RunMyJobs does not store customer data, but it generates a lot of execution metadata. Do you need all of it, or do you only care about Jobs that fail? Keep in mind that customer data must be archived on customer systems, not from within RunMyJobs.

  • Where: Your archive location should be safe, maintained, and available. Will data be stored on-site or at a different location? Follow your company standard.

  • When: How often do you need to access archived data? Is the archive intended for regular viewing, or should it only be restored when required?

The Big Picture

The image below shows the lifecycle of execution data. From left to right:

  • Process Execution: Execution data is stored in RunMyJobs.
  • Data Available in Redwood (Cloud): The execution data is available in the cloud. RunMyJobs stays clean, fast and stable by following retention rules (Keep Clauses). Any data that must be kept longer can be archived in a customer's archiving system before a data purge (for example, using Redwood Reports).
  • Data Purge: Redwood's standard housekeeping cleans up the execution data based on the defined retention.
  • Data Available in Customer's Archiving System: You will have the data in your archiving system available for long-term auditing.
  • Archive deletion: Even archiving systems will, in most cases, have retention and must be periodically cleaned up. This task can also be automated.