Score text for defamation and hate, and mask the offensive words.

Three stages, and only the first is required

Stage 1 is the core and needs no model. It normalizes the text, so spaced-out or leetspeak spellings still match, looks words up in a lexicon, splits the text into sentences, scores each sentence and masks the matches. Every span points into the original text, so masking never shifts a character.

Stage 2 is a small add-on for each language. It reads the words around a hit for cues a word list cannot hold: is the word quoted, denied, reported in the news or part of a film? Today these are rule-based baselines, not trained models.

Stage 3 is an optional LLM judge. It is off unless the caller asks for it, and it uses a model client the caller supplies. The package itself contains no model and no network code.

When the add-on is asked, and how the scores combine

Routing. A hit with a stage 1 score of 85 or more that is not ambiguous is called definite and stays at stage 1. Everything else goes to the language add-on. An add-on score that is still uncertain (between 20 and 80) can go on to the judge, if you opted in.

Fusion. The default, max, keeps the highest score, so a later stage can raise a score but never lower it. weighted averages them. veto lets the add-on lower a score when it sees a quote, a denial or a news report.

The floor. A rank 5 term never drops below 60, whatever a later stage says. A quoted slur can fall from block to review, but not to pass.

Choose the three inputs below to see what the router's own fusion code returns.

Five languages, one add-on each

Each language has its own add-on, lexicon and gold set. What makes a word abusive or merely mentioned depends on the language, so a good English result must not hide a weak Korean one.

How well it works

The chart shows F1 at the flag level (a score of at least 35) on two draft gold sets per language: a dev set the rules were tuned on, and a held-out set they were not. Both were written by an AI assistant and have had no native-speaker review. Read the numbers as a development signal, not as measured quality.

The lexicons are small, and most hate entries are long phrases that stage 1 already blocks. The gain comes from low-rank defamation terms such as "fraudster". On text from the real world these numbers would say much less.

Fast enough for comments, with a streaming mode

Stage 1 plus a language add-on takes about 0.2 ms for a comment-sized text, so one process handles thousands of messages per second. A few thousand characters take about ten milliseconds, and the cost grows in step with the length. Build the engine once per process (about 0.3 s) and reuse it.

These numbers come from short sentences of the gold sets on a 2-core development machine, most of them without any hit, and they vary from run to run. Text with many hits costs more. The judge at stage 3 is another matter: it is one model call for each uncertain sentence, so keep it off the synchronous path.

Text that arrives in pieces can be scanned with a stream scanner. It holds back only the sentence still being written, and the masked pieces it hands back add up to exactly what one scan of the whole text would give. Each sentence is scanned once, so the add-on and the judge are not called again for every chunk.

Install and run

Nothing is on PyPI yet. Clone the repository with its submodules and install the core and the add-ons you need.

git clone --recurse-submodules https://github.com/zafrem/tnscope-forge
cd tnscope-forge
pip install -e tnscope -e tnscope-en -e tnscope-ko -e tnscope-ja \
            -e tnscope-zh-hans -e tnscope-zh-hant

Add-ons are found on their own; there is nothing to import. From Python, or from the command line:

import tnscope

r = tnscope.scan("Don't trust Mike, he's a fraudster.", "en")
d = r.sentences[0].detections[0]
print(d.score, d.action, d.stage_reached)   # 60.0 review 2
print(r.masked_text)

# tnscope scan "Go back, wetback." --lang en --json
# add --max-stage 1 to turn add-ons off, --verify-definite to re-check strong hits

Stage 3 needs a model client that you provide, and runs only when you ask for it. Without one, tnscope keeps the stage 2 score and adds a warning.

import tnscope, tnscope_judge

class MyClient:
    model_id = "my-model"
    def complete(self, prompt: str, schema: dict) -> str: ...

tnscope_judge.configure(MyClient())
r = tnscope.scan("...", "en", max_stage=3)

One core, one add-on per language, one optional judge

The core holds stage 1, the router and the command line. The language add-ons and the judge are separate packages that depend on the core, never the other way round. Each is versioned, calibrated and rolled back on its own. The data (lexicons and rules) lives in its own repository and wins when documents disagree.

What is not finished

• Everything here is a draft. Scores, ranks and word lists are provisional and wait for native-speaker and policy review.
• The gold sets were written by an AI assistant. The results are a development signal, not measured quality.
• The add-ons are rule-based baselines. A quoted or negated strong slur keeps a floor of 60, so it lands in review, not in pass.
• The judge has only been tested with fake model clients, never with a real model.
• Recall is capped by the word lists. A sentence with no listed term is never passed to an add-on or to the judge, so abuse in words the lexicon lacks goes unseen at every stage.
• A strong hit (score 85 or more) is not re-checked by default, so a quoted or reported slur is blocked. Turning on verify together with veto lowers it to review.
• Nothing is released. It is not on PyPI, and no license has been chosen yet.