Flag unsupported claims

A crude hallucination check: how many words of an answer sentence appear in the source text?

What to build

A program that rates each answer sentence by its word overlap with a source and flags the weak ones.

Requirements

  • Make words by splitting on spaces, removing ., ,, ! and ? from both ends of each word, and lower-casing. Drop words that become empty.
  • The support of a sentence is the share of its words that appear anywhere in the source.
  • Print K: P% ok if the support is at least the threshold, otherwise K: P% check. P is the support as a whole percentage (:.0%).
  • Finish with unsupported U of N, where U counts the sentences marked check.

Input

Line 1: the source. Line 2: the threshold as a decimal. Line 3: n. Then n sentences.

Output

n + 1 lines.

Constraints

1 ≤ n ≤ 20. Every sentence has at least one word.

Example

Input:

The museum opens at 10 am and closes at 5 pm. Entry is free on Sundays.
0.6
3
The museum opens at 10 am.
Parking costs 3 dollars.
Entry is free on Mondays.

Output:

1: 100% ok
2: 0% check
3: 80% ok
unsupported 1 of 3

Why it works

Words that are missing from the source are a warning that the sentence may have been invented, and a threshold turns the share into a simple flag for a person to check.

Examples

Example 1 · Input
The museum opens at 10 am and closes at 5 pm. Entry is free on Sundays.
0.6
3
The museum opens at 10 am.
Parking costs 3 dollars.
Entry is free on Mondays.
Expected output
1: 100% ok
2: 0% check
3: 80% ok
unsupported 1 of 3
Example 2 · Input
cats purr
1.0
1
Cats purr loudly.
Expected output
1: 67% check
unsupported 1 of 1

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Make a word list from a text by splitting on spaces, stripping `.,!?` from both ends and lower-casing.

Hint 2

A sentence's support is the share of its words that are in the source; compare it with the threshold using `>=`.

Hint 3

Format the percentage with `f"{share:.0%}"`.

Solution

Show one correct solution (try it yourself first)
def words(text):
    return [w.strip(".,!?").lower() for w in text.split() if w.strip(".,!?")]


source = set(words(input()))
t = float(input())
n = int(input())
bad = 0
for i in range(1, n + 1):
    mine = words(input())
    share = sum(1 for w in mine if w in source) / len(mine)
    if share >= t:
        print(f"{i}: {share:.0%} ok")
    else:
        bad += 1
        print(f"{i}: {share:.0%} check")
print(f"unsupported {bad} of {n}")

Lesson: Safety and responsible use →