PrefectHQ / prefect-dbt

Collection of Prefect integrations for working with dbt with your Prefect flows.

Home Page:https://prefecthq.github.io/prefect-dbt/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Show polling status updates selectively by comparing previous status vs current status

ahuang11 opened this issue · comments

In #95 we made it debug level, but maybe we can be smarter about when to emit the logs; e.g. comparing current state to previous state


        last_state = state = current_state
        t0 = time.time()

        while state not in until_states:
            job_run = await run_sync_in_worker_thread(
                job_service_client.get_custom_job, name=full_job_name,
            )
            state = job_run.state
            if state != last_state:
                self.logger.info(f"{self._log_prefix}: State is now {state.name}.")
                last_state = state
            else:
                # Intermittently, the job will not be described. We want to respect the
                # watch timeout though.
                self.logger.debug(f"{self._log_prefix}: Job not found.")

            elapsed_time = time.time() - t0
            if timeout is not None and elapsed_time > timeout:
                raise RuntimeError(
                    f"Timed out after {elapsed_time}s while watching job for states "
                    "{until_states!r}"
                )
            time.sleep(self.job_watch_poll_interval)