A crude hallucination check: how many words of an answer sentence appear in the source text?
What to build
A program that rates each answer sentence by its word overlap with a source and flags the weak ones.
Requirements
- Make words by splitting on spaces, removing
.,,,!and?from both ends of each word, and lower-casing. Drop words that become empty. - The support of a sentence is the share of its words that appear anywhere in the source.
- Print
K: P% okif the support is at least the threshold, otherwiseK: P% check. P is the support as a whole percentage (:.0%). - Finish with
unsupported U of N, where U counts the sentences markedcheck.
Input
Line 1: the source. Line 2: the threshold as a decimal. Line 3: n. Then n sentences.
Output
n + 1 lines.
Constraints
1 ≤ n ≤ 20. Every sentence has at least one word.
Example
Input:
The museum opens at 10 am and closes at 5 pm. Entry is free on Sundays.
0.6
3
The museum opens at 10 am.
Parking costs 3 dollars.
Entry is free on Mondays.
Output:
1: 100% ok
2: 0% check
3: 80% ok
unsupported 1 of 3
Why it works
Words that are missing from the source are a warning that the sentence may have been invented, and a threshold turns the share into a simple flag for a person to check.
Examples
The museum opens at 10 am and closes at 5 pm. Entry is free on Sundays. 0.6 3 The museum opens at 10 am. Parking costs 3 dollars. Entry is free on Mondays.
1: 100% ok 2: 0% check 3: 80% ok unsupported 1 of 3
cats purr 1.0 1 Cats purr loudly.
1: 67% check unsupported 1 of 1
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
Make a word list from a text by splitting on spaces, stripping `.,!?` from both ends and lower-casing.
Hint 2
A sentence's support is the share of its words that are in the source; compare it with the threshold using `>=`.
Hint 3
Format the percentage with `f"{share:.0%}"`.
Solution
Show one correct solution (try it yourself first)
def words(text):
return [w.strip(".,!?").lower() for w in text.split() if w.strip(".,!?")]
source = set(words(input()))
t = float(input())
n = int(input())
bad = 0
for i in range(1, n + 1):
mine = words(input())
share = sum(1 for w in mine if w in source) / len(mine)
if share >= t:
print(f"{i}: {share:.0%} ok")
else:
bad += 1
print(f"{i}: {share:.0%} check")
print(f"unsupported {bad} of {n}")