LinguaSpy

What I built

• Four cities, each with its own map and language pack (French, Spanish, Japanese, Italian), and 6 missions: two in Paris, two in Madrid, one each in Tokyo and Rome.
• Conversations stored as UConversationAsset node graphs (NPC line, player challenge, player choice, listening check, end) and missions as UMissionAsset objective graphs (talk to, reach a location, collect). Both are UPrimaryDataAssets, and their IsDataValid overrides report broken graphs in the editor. No language content lives in C++.
• A sentence grader, a Leitner spaced-repetition scheduler, a scaffold policy and a fluency grade, all written as pure functions so they can be tested without a world.
• A feedback ladder: the first wrong attempt gets a hint and costs 40% of the cover penalty, the second shows the model answer and costs the rest.
• A cover meter (UActorComponent with delegates). When it reaches zero the mission stops and you practise the phrases you got wrong.
• An adaptive handler: an S or A grade on a mission gives you less translation help next time, a C gives you more.
• A versioned save format (now v4) with step-by-step migrations and async saving.
• Occlusion fade: every tick a sphere sweep runs from the camera to the player, and buildings in the way fade out: the component writes a fade value into each mesh's Custom Primitive Data, which the toon material turns into a dither. Ground and low props are skipped by comparing their bounds to the player's feet.
• Pedestrians that wander to random reachable navmesh points around a home location, and NPCs that turn to face you while you talk.
• A Slate editor tab for authoring and validating conversations.
• Python scripts that build every conversation, mission and city layout through the editor's Python API, plus offline Piper TTS voices keyed by a SHA-1 of the text and voice settings so only changed lines are regenerated. Only the French lines are voiced so far.
• A Blender Python script that builds 7 low-poly characters from primitives and binds each part rigidly to one bone of the UE mannequin skeleton, so the stock animations work. It also works around an FBX import issue that put a ×100 scale on the root bone.
• An assessment mode and a local JSONL learning log for a planned pilot study. The log is off by default and the game has no network code.

Click-to-move and the camera rig come from Unreal's top-down template. I moved the input into the game's own folders and added the occlusion fade on top.

Grading a sentence word by word

You answer a local by arranging word tiles or, once you know a phrase well, by typing it. Each challenge stores one or more accepted answers as token lists. Before comparing, every token goes through NormalizeToken: punctuation is stripped from both ends (including «», ¿, ¡ and full-width Japanese punctuation), the text is lower-cased with FText::ToLower because FString::ToLower only folds ASCII and would leave "CAFÉ" as "cafÉ", and curly apostrophes become straight ones.

Then the player's tokens are compared with each accepted answer, position by position:

const TArray<FString>& Answer = Challenge.AcceptedAnswers[AnswerIndex].Tokens;

TArray<bool> Correct;
Correct.Init(false, Player.Num());
int32 Score = 0;
for (int32 i = 0; i < Player.Num(); ++i)
{
	if (Answer.IsValidIndex(i) && Player[i] == NormalizeToken(Answer[i]))
	{
		Correct[i] = true;
		++Score;
	}
}

const bool bExact = Score == Answer.Num() && Player.Num() == Answer.Num() && Answer.Num() > 0;
// Penalize length mismatch so the closest answer is also the closest in size
Score -= FMath::Abs(Answer.Num() - Player.Num());

The score is matched positions minus the length difference, so the winning answer is also the closest in length. That answer supplies the right/wrong mark for every tile and the index of the first wrong word, which the UI highlights.

If nothing passes, the grader runs the comparison again with accents folded through a small Latin-1 and Latin Extended-A table. A match there means you knew the words and only missed an accent, so the game gives an accent hint instead of a plain "wrong". Typed input is split on whitespace including the full-width space U+3000, and Japanese uses unspaced tile chunks with an optional romaji reading on each tile.

Scheduling reviews with Leitner boxes

Every phrase you answer becomes a review item keyed <conversation path>#<node id>#<choice index>. Items live in boxes 0 to 5. A pass without help moves the item up one box. A pass after peeking at the translation, or with extra scaffolding, keeps the box and reschedules it. A fail drops it to box 1, due again in ten minutes. Times are stored as UTC ticks.

FTimespan USrsScheduler::GetInterval(int32 Box)
{
	switch (FMath::Clamp(Box, 0, MaxBox))
	{
	case 0:
	case 1:  return FTimespan::FromMinutes(10.0);
	case 2:  return FTimespan::FromDays(1.0);
	case 3:  return FTimespan::FromDays(3.0);
	case 4:  return FTimespan::FromDays(7.0);
	default: return FTimespan::FromDays(21.0);
	}
}

void USrsScheduler::Update(FSrsItemState& State, bool bPassed, bool bAssisted, const FDateTime& Now)
{
	if (!bPassed)
	{
		++State.Failures;
		State.Box = 1;
	}
	else if (bAssisted)
	{
		++State.Passes;
		++State.AssistedPasses;
	}
	else
	{
		++State.Passes;
		State.Box = FMath::Min(State.Box + 1, MaxBox);
	}
	State.LastReviewUtcTicks = Now.GetTicks();
	State.DueUtcTicks = (Now + GetInterval(State.Box)).GetTicks();
}

Review sessions take due items first, most overdue first, then lowest box, then by key so the order is deterministic. Spare slots can be filled with the weakest items that aren't due yet.

The box also decides what kind of exercise you get. A new phrase is assembled from the right tiles only; boxes 1 and 2 add distractor tiles; boxes 3 and 4 ask you to type the missing word; box 5 asks for the whole sentence. Each mistake earlier in the same session drops the level by one. The typing levels only apply when the language pack allows typed input and puts spaces between words (bAllowTypedInput && bUsesWordSpacing), so Japanese stays on tiles.

EScaffoldLevel UScaffoldPolicy::ChooseLevel(const FSrsItemState* State, int32 SessionErrors, bool bTypingAllowed)
{
	int32 Level = static_cast<int32>(EScaffoldLevel::Assemble);
	const int32 Box = State ? State->Box : 0;
	if (Box >= 5)
	{
		Level = static_cast<int32>(EScaffoldLevel::Produce);
	}
	else if (Box >= 3)
	{
		Level = static_cast<int32>(EScaffoldLevel::GapFill);
	}
	else if (Box >= 1)
	{
		Level = static_cast<int32>(EScaffoldLevel::AssemblePlus);
	}

	if (!bTypingAllowed)
	{
		Level = FMath::Min(Level, static_cast<int32>(EScaffoldLevel::AssemblePlus));
	}
	Level = FMath::Max(Level - FMath::Max(SessionErrors, 0), static_cast<int32>(EScaffoldLevel::Recognize));
	return static_cast<EScaffoldLevel>(Level);
}

The save version that never got saved

The player profile is a USaveGame subclass with CurrentVersion = 3 and a Migrate() function that upgrades one step at a time (if (Version < 2), then if (Version < 3)). The v2 to v3 step, for example, gives each city a handler mode based on an old setting and marks finished missions as already briefed so returning players don't get surprise briefings.

The bug: Version originally defaulted to CurrentVersion. Unreal's SaveGame serialization skips properties whose value equals the class default, so the version was never written to disk. On load the missing property took the class default, which is the current version, and old profiles looked up to date and skipped migration.

The fix was to make the default 0 on purpose and stamp the real version explicitly when a profile is created and every time it's saved. A regression test went in with the same commit.

/**
 *  Format version of this profile. Defaults to 0 on purpose: SaveGame serialization skips properties equal to the
 *  class default, so a default of CurrentVersion would never be written and old files would load as "current".
 *  Files written before v2 therefore load as 0 (= v1). Stamped with CurrentVersion on save / creation.
 */
UPROPERTY()
int32 Version = 0;

void ULinguaSpySaveSubsystem::RequestSave()
{
	if (Save)
	{
		Save->Version = ULinguaSpySaveGame::CurrentVersion;
		UGameplayStatics::AsyncSaveGameToSlot(Save, SlotName, 0);
	}
}

Saving is async (AsyncSaveGameToSlot), and a migration on load is logged with the old and new version numbers.

Conversation editor

The editor module adds a tab under Tools, LinguaSpy. It has an asset list with a filter and "New" and "Validate all" buttons, a list of nodes indented by their distance from the start node, and a details view for the selected node. Every edit runs inside an FScopedTransaction, so undo works. Edits made in the tab stamp a LinguaSpy.HandEdited metadata tag on the package, and the Python content scripts skip tagged assets instead of overwriting hand edits.

const FScopedTransaction Transaction(LOCTEXT("AddNode", "Add Conversation Node"));
Current->Modify();
StampHandEdited(Current);

I haven't done a visual pass on this tab yet, so expect rough layout.

Next

• Spanish and Italian voices. Tokyo stays text-only because of the voice licence.
• Two more missions (Tokyo and Rome), mission replay and per-city music.
• Testing the packaged build on another PC and a review of the content by native speakers.
• The pilot study itself. I'm not claiming any learning results.

How I built this

I designed and led LinguaSpy, from the game and learning design to the architecture and testing, and I'm happy to walk through any system on this page. I used Claude Code as an AI coding assistant, and the commits say so. The scheduler widget here is a TypeScript port written for this site.

Back to all projects