confident-ai / deepeval

The LLM Evaluation Framework

Home Page:https://docs.confident-ai.com/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

deepeval should return non-zero exitCode on failure

olensmar opened this issue · comments

the deepeval cli command currently always exits with exitCode 0 - even if the tests fail - which makes is hard to handle failed tests in automated workflows/pipelines.

For example the following workflow on GitHub ends green/successful even if the deepeval test fails.

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"

      - name: Install Dependencies
        run: pip install -U deepeval

      - name: Run DeepEval Unit Tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: deepeval test run test_example.py 

It would be great if the deepeval command could exit with a non-0 exitCode - see https://docs.github.com/en/actions/creating-actions/setting-exit-codes-for-actions

thank you!