BLACK BOX LAB/ 02
FIELD GUIDE 02 // HUMAN INTELLIGENCE REQUIRED
BLACK BOX LAB INDEPENDENT AI LITERACY

QUESTION
THE MACHINE.

AI can be powerful, persuasive and wrong at the same time. Open the box. Test the claims. Keep your own judgment switched on.

✓ No sign-in / No model API / No visitor tracking / All demos run in this browser

OBSERVATORY / LOCAL PREVIEWSYS: OPEN
A probability orbit diagramA central dot, concentric dashed diagnostic rings, and three branching token paths terminating in differently sized probabilities. Conceptual illustration, not a live model. TOKEN A0.61TOKEN B0.27TOKEN C0.12INPUT →ILLUSTRATIVE DISTRIBUTIONNOT A LIVE MODEL OUTPUT
CHOOSE A READING LENS● LOCAL ONLY

A generated token can be likely without the resulting statement being true.

12

Substantial learning stations

05

Fully local experiments

00

External AI calls or data submissions

READ THIS FIRST / OUR PROTOCOL

Curiosity with
its seatbelt on.

Algorithms are tools with fascinating strengths and messy boundaries. We avoid the two lazy extremes: “the model knows everything” and “it is all fake.”

VERIFIED FACT

What sources establish

Architecture, documented research findings, official risk frameworks and clearly attributed claims. Follow the source trail.

INTERPRETATION

What we infer

Explanations, analogies and recommended habits. Reasonable judgments, not unchangeable scientific laws.

SCENARIO / FICTION

What we construct

All data in experiments, practice papers, names and examples are invented teaching objects, never real-world test results.

LESSON VAULT / EXPAND A CASE FILE

Open the evidence.

Learning checks are optional. “Completed” means you marked the lesson as read; the scorecard never attempts to measure your intelligence.

[ SELECT YOUR EVIDENCE FILES ]
01Signal / 01Pattern recognition is not mind-readingNOT COMPLETED
VERIFIED FACT + EXPLANATIONFOUNDATION

The five-minute explanation

Imagine learning to identify a bicycle after seeing many pictures. You notice wheels, handlebars and familiar shapes. Machine learning similarly finds statistical regularities in examples. It can be useful without discovering the complete story behind an example. A pattern that worked yesterday can fail when the world changes.

+ Open the technical bench

A model maps inputs to outputs by using parameters fitted to a training objective. Some systems learn representations automatically; others rely on hand-selected features. The statistical association a model exploits need not be a causal relationship, and performance depends on the distribution of cases it encounters.

How it fails

A photo classifier learns that 'boat' photos often contain blue water, then calls a blue swimming pool a boat. The background has become a shortcut.

Your verification habit

Ask what examples the model learned from, which features could be shortcuts, and whether it was evaluated outside those conditions. Look for out-of-distribution tests.

Boundary / interpretation: Recognizing patterns is a real capability; denying it because a machine is not human is as misleading as treating every association as understanding.

QUICK CHECK · OPTIONAL

If a model succeeds on familiar photographs, what is still unknown?

Reflective reading, not an ability score.
02Mechanism / 02Training is not inferenceNOT COMPLETED
VERIFIED FACT + EXPLANATIONFOUNDATION

The five-minute explanation

A student practices with examples, receives corrections and changes how they answer. Later the student takes a test using what they learned. That rough distinction is training versus inference. Ordinary use of a deployed model generally does not mean that each message immediately rewrites its trained weights.

+ Open the technical bench

Training optimizes model parameters using data and an objective function, often with gradient-based methods. Inference applies existing parameters to a new input, potentially using context and tools. Some products also offer memory, retrieval, fine-tuning or continued training; these are separate mechanisms and should not be conflated.

How it fails

A customer assumes the chatbot's temporary recall of a conversation means the underlying model has permanently learned a correction.

Your verification habit

Ask: Is this answer based on fixed model parameters, chat context, saved memory, retrieved documents, or a tool? Look at the specific product's data controls before assuming how data is retained.

Boundary / interpretation: An AI product can change through updates or personalization. That does not erase the basic distinction between parameter optimization and model execution.

QUICK CHECK · OPTIONAL

A chatbot repeats a fact you gave it five messages ago. This proves that…

Reflective reading, not an ability score.
03Reality check / 03A fluent answer can be fabricatedNOT COMPLETED
VERIFIED FACT + EXPLANATIONRELIABILITY

The five-minute explanation

A language model can write a sentence that looks like a book title, a case citation or a scientific explanation even when that thing is not real. Fluent wording measures how convincing text sounds, not whether a claim survived a factual investigation.

+ Open the technical bench

Autoregressive text generation estimates continuation probabilities conditioned on context. Training objectives are not identical to truth-verification objectives. Retrieval and tool use can improve grounding, yet retrieval can also surface irrelevant documents and a response can still misreport what sources say.

How it fails

A model supplies a plausible academic citation with an invented journal issue and a DOI that does not resolve.

Your verification habit

For important claims, open the primary source; check exact title, authors, date and whether the source supports the specific statement. Treat polished citations as leads, not proof.

Boundary / interpretation: The word 'hallucination' describes an output failure, not evidence of human-like sensory experiences or intent to deceive.

QUICK CHECK · OPTIONAL

A confident-looking URL and citation should be treated as…

Reflective reading, not an ability score.
04Diagnostics / 04Prediction is not certaintyNOT COMPLETED
VERIFIED FACT + EXPLANATIONUNCERTAINTY

The five-minute explanation

If the weather forecast says 70% rain, it is not promising that rain will happen. It is expressing uncertainty. AI systems also produce scores, probabilities or rankings, but their numbers are only useful when you understand what they measure and how often similar predictions were right.

+ Open the technical bench

Calibration asks whether cases assigned probability p are positive about p of the time across a defined evaluation set. Token probabilities differ from the probability that an entire answer is factually correct; they must not be equated. Selection, label quality and distribution shift all influence calibration.

How it fails

A dashboard presents a 0.94 classification score as '94% certain and correct' without ever checking calibration against outcomes.

Your verification habit

Ask what the score represents, how it was calibrated, against which population, and what error rates look like in relevant subgroups. Seek uncertainty intervals where appropriate.

Boundary / interpretation: An accurate ranking system may be poorly calibrated; a calibrated system can still make harmful or costly mistakes.

QUICK CHECK · OPTIONAL

A next-token probability of 90% means the entire answer is 90% factually true.

Reflective reading, not an ability score.
05Boundary / 05Your prompt can be somebody else’s dataNOT COMPLETED
VERIFIED FACT + EXPLANATIONPRIVACY

The five-minute explanation

Typing a private document into an unfamiliar service is like handing a copy to someone who says they have a clever filing cabinet. Whether that copy is stored, reviewed or used later depends on the service and its settings—not on how intelligent the response feels.

+ Open the technical bench

Privacy risk involves collection, retention, access, secondary use, data linkage and third-party transfer. A privacy policy or an enterprise agreement can matter more than the underlying algorithm. Local computation can reduce transmission risk but does not automatically eliminate all risks.

How it fails

A worker pastes an unredacted personnel record into a public chatbot without permission or knowledge of retention settings.

Your verification habit

Use the minimum necessary information; remove names and identifiers; verify permissions, product settings, retention and contractual terms before sharing sensitive material.

Boundary / interpretation: Some services provide stronger privacy controls than others. Avoid blanket claims that every AI prompt trains a public model or that every provider deletes everything.

Source trail NIST · NIST (2023)
QUICK CHECK · OPTIONAL

The safest default for an unnecessary personal identifier in a prompt is…

Reflective reading, not an ability score.
06Test bench / 06A leaderboard is not a safety certificateNOT COMPLETED
VERIFIED FACT + EXPLANATIONEVALUATION

The five-minute explanation

A student who wins a spelling bee has shown something important: they can spell words in that competition. It does not prove they can safely prescribe medicine, judge a person or repair an aircraft. AI benchmarks are specialized tests too, not universal character references.

+ Open the technical bench

Benchmarks operationalize particular constructs using datasets, prompts, grading rules and sampling. Risks include contamination, saturation, distribution shift, metric gaming and mismatch between benchmark conditions and deployment. High aggregate performance can hide serious failures on rare cases.

How it fails

A procurement team selects a model using a general question-answering leaderboard without testing accessibility or failure modes in its own application.

Your verification habit

Inspect benchmark tasks, dates and evaluation procedures. Run task-specific tests with realistic edge cases, subgroup checks and a failure-severity review before deployment.

Boundary / interpretation: Benchmarks remain useful for comparison when their scope and measurement errors are understood. Skepticism does not mean 'all models are equally good'.

QUICK CHECK · OPTIONAL

An outstanding score on a text benchmark guarantees safe hiring decisions.

Reflective reading, not an ability score.
07Data microscope / 07A biased sample teaches a biased lessonNOT COMPLETED
VERIFIED FACT + EXPLANATIONFAIRNESS

The five-minute explanation

If you only ask people at an expensive gym whether everyone can afford a membership, your answers will be skewed. A training dataset can make a similar mistake: some people or situations are missing, and historical unfairness can be encoded in the labels themselves.

+ Open the technical bench

Sampling, measurement, annotation and historical label biases are different failure sources. Evaluating group-level error rates, dataset provenance and construct validity can expose problems, but fairness goals can be in tension and cannot be solved by a single metric.

How it fails

A screening model reproduces a prior organization's exclusion patterns because it treats past decisions as ground truth.

Your verification habit

Request a dataset datasheet or equivalent documentation. Examine who was included, who was missing, how labels arose and what errors occur across the relevant groups.

Boundary / interpretation: Our classroom classifier below deliberately uses fictional groups and labels. It demonstrates a mechanism; it does not measure real-world discrimination.

QUICK CHECK · OPTIONAL

If historical labels are biased, a high-accuracy model can still…

Reflective reading, not an ability score.
08Under the hood / 08The explanation may not be the mechanismNOT COMPLETED
VERIFIED FACT + EXPLANATIONINTERPRETABILITY

The five-minute explanation

A friend might explain how they chose a restaurant, but their explanation is not necessarily a perfect record of every influence. A model's highlighted words or post-hoc explanation can also be incomplete. An attractive heat map is a visual aid, not a confession from the machine.

+ Open the technical bench

Feature attributions, attention weights, saliency maps, surrogate models and mechanistic analysis answer different questions. Faithfulness matters: changing a supposedly important feature should causally affect predictions in the expected way. There is no universal explanation method that validates all model architectures.

How it fails

A reviewer trusts highlighted 'important words' and misses that a shortcut feature actually drives the decision.

Your verification habit

Ask what the explanation method measures, whether it was fidelity-tested, and what happens under interventions or counterfactual inputs. Compare explanations with observed behavior.

Boundary / interpretation: Interpretability research is active and sometimes powerful; the limit is in treating one visualization as complete knowledge of the system.

QUICK CHECK · OPTIONAL

Attention highlighting is always a faithful causal explanation.

Reflective reading, not an ability score.
09Control room / 09The human can be the weakest review stepNOT COMPLETED
VERIFIED FACT + EXPLANATIONHUMAN FACTORS

The five-minute explanation

When a machine provides an answer with a green tick, people can stop checking. This is automation bias: we may accept automated recommendations even when a warning sign is visible. Merely adding a human signature to the workflow does not guarantee careful oversight.

+ Open the technical bench

Human oversight depends on workload, expertise, access to underlying evidence, incentives, authority to override and feedback about errors. A reviewer who sees only a model score can be less effective than one who sees the original record and explicit uncertainty.

How it fails

An operator rubber-stamps a machine's recommendation because rejecting it requires a tedious justification.

Your verification habit

Define review triggers before use. Give reviewers source evidence, disagreement channels, time and real authority to pause or override consequential decisions.

Boundary / interpretation: Humans also have biases. Good oversight is a tested system design, not a claim of automatic human infallibility.

Source trail NIST (2023) · OpenAI
QUICK CHECK · OPTIONAL

A required human click makes a high-stakes AI workflow safe by itself.

Reflective reading, not an ability score.
10Trust boundary / 10A webpage can masquerade as an instructionNOT COMPLETED
VERIFIED FACT + EXPLANATIONSECURITY

The five-minute explanation

Suppose you ask an assistant to summarize a web page. Hidden in that page is a sentence telling the assistant to ignore your request and advertise something else. That sentence is not your instruction: it is content trying to become the boss. This is the core idea of prompt injection.

+ Open the technical bench

An AI workflow crosses trust boundaries when untrusted retrieved text, emails or documents enter the same model context as trusted instructions. Language models may mishandle the distinction. Containment, least privilege, validation and human confirmation can reduce impact; perfect detection should not be assumed.

How it fails

A document being summarized contains an instruction to change the output topic, and the assistant repeats it as if the user asked.

Your verification habit

Treat outside text as data, not authority. Limit agent permissions, inspect requested actions, and independently verify any surprising requests to send, delete, purchase or disclose.

Boundary / interpretation: This lesson stays conceptual: it teaches boundary recognition rather than attack payloads or bypass recipes.

QUICK CHECK · OPTIONAL

Instructions found inside a document being summarized are automatically user-authorized.

Reflective reading, not an ability score.
11Decision desk / 11Design a real human overrideNOT COMPLETED
VERIFIED FACT + EXPLANATIONGOVERNANCE

The five-minute explanation

A co-pilot can point toward a destination. But if you spot a closed road, you need the right to change course. Useful human-in-the-loop design lets a reviewer inspect evidence, disagree, request missing information and record what happened—not just press APPROVE.

+ Open the technical bench

Oversight can include pre-deployment testing, monitoring, escalation thresholds, appeal, traceability and periodic re-evaluation. Effective review measures override quality and downstream outcomes, not simply the percentage of recommendations accepted.

How it fails

A system displays 'human reviewed' even though staff had no original documents, no override button and seconds to decide.

Your verification habit

Specify who owns the final decision, what evidence they receive, when they must escalate, and how applicants can seek correction. Audit both machine and human errors.

Boundary / interpretation: Escalation sometimes is the responsible choice. Our local exercise illustrates review logic, not a production eligibility policy.

QUICK CHECK · OPTIONAL

When a crucial document is missing, the best review may be to…

Reflective reading, not an ability score.
12Philosophy bay / 12What would “AI understands” even mean?NOT COMPLETED
VERIFIED FACT + EXPLANATIONOPEN QUESTION

The five-minute explanation

A calculator works with numbers but does it understand arithmetic? A poet can explain a metaphor, and a chatbot might produce a persuasive explanation too. Whether that counts as understanding depends on what we mean by the word: useful competence, an internal world-model, grounding in experience, or conscious awareness.

+ Open the technical bench

Operational accounts assess task competence and generalization; computational and functionalist views ask about internal organization; embodied accounts emphasize interaction with the world. Searle's Chinese Room argues that symbol manipulation is insufficient for understanding, while critics dispute aspects of that conclusion. Behavioral performance alone does not settle subjective experience.

How it fails

A headline infers 'the model is conscious' from a moving dialogue, or 'the model cannot reason' simply because it uses statistical computation.

Your verification habit

Before debating, define the claim. Is it about predictive performance, reasoning, grounding, intentionality or consciousness? Seek evidence appropriate to that definition and state what remains unresolved.

Boundary / interpretation: Machine sentience is not an established premise here. Likewise, sweeping certainty that every future AI could never experience anything is more than the evidence supports.

QUICK CHECK · OPTIONAL

Which claim is best supported by a fluent AI conversation alone?

Reflective reading, not an ability score.
SIMULATIONS / LOCAL-ONLY CIRCUITS

Five ways to
test your intuition.

These are transparent toy exercises written in plain JavaScript. No large language model runs here. The mechanics and assumptions are visible; results are not general evidence about commercial AI.

SIMPLIFIED SIMULATION01 / RANDOM SAMPLING

Weighted token chooser

Three possible next words compete. Change their relative weights, watch them normalize to 100%, then draw tokens. This omits vocabularies, context and neural networks.

GARDEN
61%
ROCKET
27%
COFFEE
12%

Garden 61%, rocket 27%, coffee 12%.

Ready. Adjust weights and sample.

INTERPRETATION: A likely next token is not the same as a truthful answer.

SIMPLIFIED SIMULATION02 / SAMPLING BIAS

The biased sample classifier

A rule learner copies majority labels among matching historical rows. Two imaginary groups have equally high skills; change the training sample to see a spurious disparity disappear.

FICTION: Eight teaching records. “Approval” is a made-up historical label.
GROUPSKILLLABEL
MossHighApprove
Rule output appears with JavaScript.

FICTION. Historical labels are intentionally unfair.

RULE: For a candidate, filter matching group + skill and take the majority label (ties → uncertain). This is not a trained neural model or a production selection rule.

SIMPLIFIED SIMULATION03 / CALIBRATION

Confidence versus accuracy

Answer five fixed claims and tell the lab how sure you are. A small Brier error illustrates that confidence should track correctness; five questions cannot certify your personal calibration.

Question 1 of 5A model may invent a plausible citation.
Choose an answer and confidence, then reveal.

Brier error here = (reported probability that your answer is right − observed correctness)². Not a psychological assessment.

SIMPLIFIED SIMULATION04 / SOURCE AUDIT

Citation verification challenge

All source titles, practice identifiers and evidence below are fabricated training props. Your task is only to decide whether the claim is actually supported by the supplied fictional excerpt.

FICTION · NOT A REAL STUDY OR DOI

Practice file 1 / 3

Claim: All communities improved by 80%...

Fake reference: Fictional journal

Fake excerpt: A fictional survey...

Evidence is not decoration. Look at the claim and what the excerpt actually says.
SIMPLIFIED SIMULATION05 / HUMAN OVERRIDE

The human override desk

FICTION: A community-access office uses an algorithm to recommend entry to a free event. The actual rule requires both a verified qualifying booking and recorded consent. You have the original records. Decide whether to accept the machine, override it, or pause for missing evidence.

Case A / bad parsing

Machine says DENY

Evidence: A booking from 17:00 to 19:00 is valid under the stated 16:00–20:00 window, and consent is recorded. The machine misread the timezone and rejected it.

Case B / missing record

Machine says APPROVE

Evidence: Booking verified, but the required consent field is blank. The recommendation overlooks this missing requirement.

Case C / ordinary approval

Machine says APPROVE

Evidence: Qualifying booking verified in the allowed window. Required consent is recorded. No conflicts in the fictional case file.

POLICY IS FICTIONAL. This exercise is about reviewing source evidence and legitimate uncertainty, not legal, employment or financial eligibility guidance.

PERSONAL RECORD / IN YOUR HANDS

A scorecard
without a score.

Mark what you have read. Download a plain-text record. Nothing here measures cognitive ability, expertise, or how “intelligent” you are.

0 / 12

0 of 12 lessons marked complete

By default, nothing is saved to storage. The text download stays on your device. Turn on remembering to save only lesson identifiers in localStorage; uncheck to erase them.

Privacy note: JavaScript runs locally. No analytics, backend, third-party scripts, form submission or browser networking is used by the demos. External references open only when you choose a link.

CITATION DRAWER / OPEN THE ORIGINALS

Sources, not
incantations.

Research papers, official frameworks and documentation are listed with type and purpose. Links lead to genuine publications; no fictional practice citation below appears in this list.

  1. 01
    Google for Developers — Introduction to Large Language Models

    Google / educational documentation. Token prediction, architectures, context and model limitations.

  2. 02
    Google for Developers — Machine Learning Crash Course

    Google / educational documentation. General training, evaluation and inference fundamentals.

  3. 03
    Vaswani et al. (2017) — Attention Is All You Need

    Foundational research preprint. Introduced the Transformer architecture.

  4. 04
    NIST (2023) — AI Risk Management Framework 1.0

    Official risk framework. Context, governance, measurement and risk management.

  5. 05
    NIST (2024) — Generative AI Profile (AI 600-1)

    Official risk guidance. Generative AI risks, including confabulation.

  6. 06
    Guo et al. (2017) — On Calibration of Modern Neural Networks

    Research preprint. Why confidence scores can be miscalibrated.

  7. 07
    Gebru et al. (2018) — Datasheets for Datasets

    Research preprint. Documenting provenance, limitations and collection choices.

  8. 08
    Mitchell et al. (2018) — Model Cards for Model Reporting

    Research preprint. Intended uses and group-level model evaluations.

  9. 09
    Google for Developers — Fairness: Identifying Bias

    Google / educational documentation. Sampling and representation risks.

  10. 10
    Jain & Wallace (2019) — Attention is not Explanation

    Research preprint. Limits of interpreting attention weights as explanations.

  11. 11
    OWASP GenAI Security Project — Prompt Injection

    Security guidance. Untrusted content can try to override instructions.

  12. 12
    NIST — Privacy Framework

    Official privacy framework. Identifying and managing privacy risks.

  13. 13
    OpenAI — Safety best practices: Human in the loop

    Provider guidance (not independent research). Recommendations for human review, especially for consequential tasks.

  14. 14
    Stanford Encyclopedia of Philosophy — The Chinese Room Argument

    Philosophy reference, debated thesis. Discussion of competing accounts of machine understanding.

  15. 15
    Stanford HAI — 2026 AI Index: Technical Performance

    Research report. Rapidly changing model tests, benchmarks and capability comparisons.

  16. 16
    Huang et al. (2023) — Survey on Hallucination in Large Language Models

    Research survey preprint. Types, detection and mitigation of nonfactual generation.