Metamorphic Testing

Giriş

Konuyu ilk defa burada gördüm.

Traditional testing assumes you know the expected output.

assertEquals(4, calculator.add(2, 2));

But how do you test an LLM that generates open-ended answers when there is no single correct output?

This is called the oracle problem: determining whether a program's output is correct when the expected answer is difficult or impossible to establish.

Metamorphic testing addresses this by checking relationships between multiple executions instead of checking each output against a known answer.

Örnek

How metamorphic testing works

The key concept is a metamorphic relation (MR): a property that should remain true when you transform an input and execute the system again.

1. Original input

A woman is sitting on a park bench with her dog.

Paraphrase without changing meaning

2. Transformed input

A woman sits outdoors with her pet nearby.

Run both inputs through the LLM

Same classification : Relation holds

Different classification : Potential fault

The important point is that you don't need to know the correct classification in advance. If two semantically equivalent inputs produce contradictory classifications, the model may be behaving inconsistently.