Getting Started with Prefect

This guide will walk you through installing Prefect, writing your first workflow, and deploying it for production use.

Time: 45-60 minutes Prerequisites: Python 3.8+, basic Python knowledge, pip or conda

---

Installation

Option 1: pip (Recommended)

Option 2: conda

Option 3: Docker

---

Choose Your Path: Cloud or Self-Hosted

Option A: Prefect Cloud (Recommended for Getting Started)

Advantages:

• No infrastructure setup
• Instant start
• Free tier (20,000 task runs/month)
• Built-in monitoring and UI

Setup:

Option B: Self-Hosted Server

Advantages:

• Full control
• No external dependencies
• Unlimited usage
• Air-gapped deployment possible

Setup:

Access UI: http://127.0.0.1:4200

---

Your First Flow

Step 1: Write a Simple Flow

Create hello_prefect.py:

Step 2: Run Locally

Output:

You'll see logs in your terminal and the run tracked in the Prefect UI!

---

Your First Data Pipeline

Step 1: Install Additional Dependencies

Step 2: Create an ETL Pipeline

Create data_pipeline.py:

Step 3: Run the Pipeline

What Happened:

• ✅ Data extracted with automatic retries on failure
• ✅ Results cached for 1 hour (re-run immediately to see caching in action)
• ✅ Each task logged and tracked
• ✅ Full run history visible in UI
• ✅ Output file created

---

Add Error Handling and Logging

Enhance your pipeline with better error handling:

---

Deploy Your First Flow

Step 1: Create a Deployment

Step 2: Customize the Deployment (Optional)

Edit etl-deployment.yaml to add a schedule:

Step 3: Apply the Deployment

Step 4: Start a Worker

---

Working with Parameters

Define Parameterized Flows

Run Deployments with Parameters

---

Using Blocks for Configuration

Blocks store reusable configuration and credentials.

Create a Block (UI Method)

1. Go to Prefect UI → Blocks
2. Click "+" to create new block
3. Choose "Secret" or "JSON" or database connection
4. Save with a name

Create a Block (Code Method)

Use Blocks in Flows

---

Essential CLI Commands

Flow Management

Deployment Management

Work Pool and Worker Management

Monitoring

---

Development Workflow

Best Practice Development Loop

1. Write flow locally in Python file
2. Test execution with python your_flow.py
3. Iterate until it works correctly
4. Create deployment with prefect deploy
5. Start worker to execute deployed flows
6. Monitor in Prefect UI
7. Adjust and redeploy as needed

Project Structure

---

Testing Flows

Unit Test Example

Run Tests

---

Troubleshooting

Issue: "No module named 'prefect'"

Solution:

Issue: Cannot Connect to Prefect API

Solution:

Issue: Worker Not Picking Up Runs

Solution:

Issue: Flow Runs But Not Visible in UI

Solution:

• Ensure you're connected to correct API (cloud vs local)
• Check that Prefect server/cloud is accessible
• Verify API key is valid (for cloud)

---

Next Steps

Immediate (Day 1)

1. ✅ Complete Tutorial 1: Build a Complete ETL Pipeline
2. ✅ Create your first deployment with a schedule
3. ✅ Set up a worker and run a scheduled flow
4. ✅ Explore the Prefect UI

Short-term (Week 1)

1. Read Best Practices
2. Implement error handling and retries
3. Create and use Blocks for credentials
4. Set up notifications (Slack, email)
5. Explore Use Cases

Medium-term (Month 1)

1. Migrate an existing pipeline to Prefect
2. Implement comprehensive logging
3. Set up CI/CD for deployments
4. Configure work pools for production
5. Implement monitoring and alerting

---

Quick Reference

Essential Decorators

Common Patterns

---

Resources

• Official Documentation
• Prefect Recipes - Code examples
• Slack Community - Get help
• Discourse Forum - Discussions

---

Ready to build production pipelines? → See Best Practices

Want real-world examples? → Explore Use Cases

Hands-on learning? → Start Tutorial 1

---

← Back to Prefect Overview | Next: Use Cases →