Technical Details - Jobs

General Architecture

Compliance Jobs

1. Scheduling Jobs:

   • The Scheduler Service checks every 60 seconds to identify jobs that need scheduling.

   • Jobs are recorded in PostgreSQL as "Queued" and written to NATS.

2. Job Execution Rules:

   • Each control per account per benchmark constitutes one job.

   • If a control is shared between two root benchmarks, it is triggered twice.

   • If a control is shared between two benchmarks under the same root, it is triggered once.

3. Job Completion and Summarization:

   • After job execution, the Summarizer is triggered to generate a compliance summary.

   • The job status in PostgreSQL is updated to "Summarizer in progress."

4. Recording Findings:

   • Findings are generated per worker and recorded in the database.

5. Accessing Summary Data:

   • Summary data becomes available after the Summarizer has processed all jobs.

Discovery works

1. Scheduling: Every 60 seconds, the Scheduler Service checks for jobs that need scheduling. It marks the jobs as "Queued" in PostgreSQL and notifies NATS.

2. Job Distribution: NATS workers continuously poll NATS to determine which jobs need execution.

3. Resource Management: KEDA monitors the NATS queue and scales Describer Workers accordingly. Each Describer Worker is specific to a resource type and account and uses encrypted credentials to operate.

4. Data Handling: Describer Workers decrypt credentials, execute tasks, and log results to Elasticsearch.

5. Data Summarization: The Summarizer processes data for each account every 24 hours, generating summarized metrics from the discovery workers.