marvin.utilities.testing
Utilities for running unit tests.
assert_equal
¶
Asserts whether the LLM output meets the expected output.
This function uses an LLM to assess whether the provided output (llm_output) meets some expectation. It allows us to make semantic claims like "the output is a list of first names" to make assertions about stochastic LLM outputs.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
llm_output |
Any
|
The output from the LLM. |
required |
expected |
Any
|
The expected output. |
required |
Returns:
Name | Type | Description |
---|---|---|
bool |
bool
|
True if the LLM output meets the expectation, False otherwise. |
Raises:
Type | Description |
---|---|
AssertionError
|
If the LLM output does not meet the expectation. |