mirror of
https://github.com/vinta/awesome-python.git
synced 2026-11-10 01:34:28 -06:00
docs: add temporalio to Job Schedulers intro
The intro did not mention a README entry added after it was written, so readers got no advice on it. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -7,6 +7,7 @@ How to choose:
|
||||
- Your Python functions as pipelines, with tasks created at runtime: Prefect
|
||||
- Pipelines built around the data assets they produce, like tables and models: Dagster
|
||||
- A simple loop of periodic jobs in one script: schedule
|
||||
- Long-running business processes that survive crashes, with retries: Temporal
|
||||
|
||||
APScheduler [schedules your Python code to run later](https://github.com/agronholm/apscheduler/blob/3.x/README.rst), once or periodically. It runs inside your existing application, not as a service. Give each job [a trigger](https://apscheduler.readthedocs.io/en/latest/userguide.html#choosing-the-right-scheduler-job-store-s-executor-s-and-trigger-s): `date` runs it once, `interval` at fixed intervals, and `cron` at set times of day. Jobs [live in memory by default](https://apscheduler.readthedocs.io/en/latest/userguide.html#basic-concepts). When they must survive restarts and crashes, add a persistent job store.
|
||||
|
||||
@@ -18,4 +19,6 @@ Prefect [turns your Python functions into data pipelines](https://docs.prefect.i
|
||||
|
||||
Dagster is [a data orchestrator built for data engineers](https://docs.dagster.io/), with lineage and observability built in. You [declare data assets like tables, datasets, and ML models as Python functions](https://github.com/dagster-io/dagster). Dagster runs them at the right time to keep them up to date. If you're just starting out, its docs [strongly recommend assets rather than ops](https://docs.dagster.io/guides/build/ops). Run assets on [a cron schedule](https://docs.dagster.io/guides/automate/schedules).
|
||||
|
||||
In an orchestrator, make every task safe to run twice. Airflow [can retry a failed task](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#creating-a-task), so its docs say a task should produce the same outcome on every re-run. Prefect tasks are [retryable units of work](https://docs.prefect.io/v3/concepts/tasks) too.
|
||||
Temporal [runs your application code as a Durable Execution](https://docs.temporal.io/evaluate/understanding-temporal#durable-execution). Once a workflow starts, it runs to completion, even when that takes a year. If its process crashes, another process picks it up where it stopped. With the Temporal Python SDK, [put `@activity.defn` on each function that does one action, like calling an API, and `@workflow.defn` on the class that orchestrates them](https://docs.temporal.io/develop/python/set-up-your-local-python#run-hello-world-test-your-installation). Keep anything that talks to the outside world in an activity, since Temporal [reruns workflow code to rebuild its state](https://docs.temporal.io/evaluate/understanding-temporal#workflows). A failed activity [retries with backoff](https://docs.temporal.io/evaluate/understanding-temporal#activities). [A worker on your own infrastructure](https://docs.temporal.io/evaluate/understanding-temporal#workers) runs your code and asks the Temporal Service for work. [The Temporal Service is the Temporal Server plus a database](https://docs.temporal.io/temporal#temporal-platform), and you run it yourself or use Temporal Cloud.
|
||||
|
||||
In an orchestrator, make every task safe to run twice. Airflow [can retry a failed task](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#creating-a-task), so its docs say a task should produce the same outcome on every re-run. Prefect tasks are [retryable units of work](https://docs.prefect.io/v3/concepts/tasks) too. Temporal [recommends idempotent activities](https://docs.temporal.io/activity-definition#idempotency), since a retried activity can run more than once.
|
||||
|
||||
Reference in New Issue
Block a user