René's URL Explorer Experiment


Title: Inspect AI, An OSS Python Library For LLM Evals – Hamel’s Blog

Open Graph Title: Inspect AI, An OSS Python Library For LLM Evals – Hamel’s Blog

X Title: Inspect AI, An OSS Python Library For LLM Evals – Hamel’s Blog

Description: A look at Inspect AI with its creator, JJ Allaire.

Open Graph Description: A look at Inspect AI with its creator, JJ Allaire.

X Description: A look at Inspect AI with its creator, JJ Allaire.

X: @HamelHusain

Generator: quarto-1.10.19

direct link

Domain: hamel.dev

authorHamel Husain
dcterms.date2025-06-23
quarto:offset../../../
msvalidate.01F9BFAF34FB8220973415C67CA60EB1A0
og:typearticle
og:imagehttps://hamel.dev/notes/llm/evals/inspect_images/inspect_cover.png
og:site_nameHamel’s Blog
og:localeen_US
og:image:height720
og:image:width1280
twitter:imagehttps://hamel.dev/notes/llm/evals/inspect_images/inspect_cover.png
twitter:creator@HamelHusain
twitter:cardsummary_large_image
twitter:image-height720
twitter:image-width1280

Links:

Join 5,000+ engineers & PMs in mastering AI Evals. Next cohort Jan 30. Learn More →https://maven.com/parlance-labs/evals?promoCode=hamel-dev
Bloghttps://hamel.dev/index.html
Noteshttps://hamel.dev/notes/index.html
OSShttps://hamel.dev/oss/opensource.html
Teachinghttps://hamel.dev/talks.html
https://hamel.dev/notes/llm/evals/inspect.html
Noteshttps://hamel.dev/notes/index.html
LLMshttps://hamel.dev/notes/llm/index.html
AI evals resourceshttps://hamel.dev/notes/llm/evals/index.html
Inspect AI, An OSS Python Library For LLM Evalshttps://hamel.dev/notes/llm/evals/inspect.html
Noteshttps://hamel.dev/notes/index.html
Python Concurrencyhttps://hamel.dev/notes/concurrency.html
CUDA Version Managementhttps://hamel.dev/notes/cuda.html
GitHub Actionshttps://hamel.dev/notes/actions/index.html
dbthttps://hamel.dev/notes/dbt/index.html
Dockerhttps://hamel.dev/notes/docker/index.html
How to learnhttps://hamel.dev/notes/how-to-learn/index.html
pandoc filtershttps://hamel.dev/notes/pandoc/index.html
programming languageshttps://hamel.dev/notes/programming-languages/index.html
Video Editinghttps://hamel.dev/notes/video_editing.html
Coding Agentshttps://hamel.dev/notes/coding-agents/index.html
Amphttps://hamel.dev/notes/coding-agents/amp.html
fastaihttps://hamel.dev/notes/fastai/index.html
Fundamentalshttps://hamel.dev/notes/fastai/fundamentals.html
Image Classificationhttps://hamel.dev/notes/fastai/cv.html
Datahttps://hamel.dev/notes/fastai/data.html
FastHTMLhttps://hamel.dev/notes/fasthtml/index.html
Building Annotation Apps with FastHTMLhttps://hamel.dev/notes/fasthtml/annotation-apps.html
Concurrency For Starlette Apps (e.g FastAPI / FastHTML)https://hamel.dev/notes/fasthtml/concurrency.html
Jupyterhttps://hamel.dev/notes/jupyter/index.html
Fix Jupyter CUDA cachehttps://hamel.dev/notes/jupyter/Fix Jupyter CUDA cache.html
K8shttps://hamel.dev/notes/k8s/index.html
Basicshttps://hamel.dev/notes/k8s/02-Basics.html
Secretshttps://hamel.dev/notes/k8s/03-Secrets.html
Storagehttps://hamel.dev/notes/k8s/storage/index.html
Storage Basicshttps://hamel.dev/notes/k8s/storage/04-Basics.html
Dynamic Provisioninghttps://hamel.dev/notes/k8s/storage/05-Dynamic Provisioning.html
Scalinghttps://hamel.dev/notes/k8s/scaling/index.html
ReplicaSetshttps://hamel.dev/notes/k8s/scaling/06-ReplicaSets.html
Scalinghttps://hamel.dev/notes/k8s/scaling/07- Scaling.html
StatefulSethttps://hamel.dev/notes/k8s/12-StatefulSet.html
Jobs & CronJobshttps://hamel.dev/notes/k8s/13-JobsCron.html
Rolloutshttps://hamel.dev/notes/k8s/14-RolloutsRollbacks.html
Helmhttps://hamel.dev/notes/k8s/helm/index.html
Creating Helm Chartshttps://hamel.dev/notes/k8s/helm/16-Creating Your Own Helm Charts.html
Testing With Helmhttps://hamel.dev/notes/k8s/helm/21-Testing-With-Helm.html
Multi-Container Podshttps://hamel.dev/notes/k8s/multi_container_pods/index.html
Multi-Container Podshttps://hamel.dev/notes/k8s/multi_container_pods/08-Multi-Container-Pods.html
Ambassador Sidecarshttps://hamel.dev/notes/k8s/multi_container_pods/09-Ambassador Sidecars.html
Restart Conditionshttps://hamel.dev/notes/k8s/multi_container_pods/10-Downsides of MC Pods.html
Sharing Processes in MC Podshttps://hamel.dev/notes/k8s/multi_container_pods/11-Sharing Processes in MC Pods.html
Developer tipshttps://hamel.dev/notes/k8s/18-Developer.html
Pod restart vs. replacementhttps://hamel.dev/notes/k8s/19-Pod-Lifecycle.html
Probeshttps://hamel.dev/notes/k8s/20-Health-Check.html
Resource Limitshttps://hamel.dev/notes/k8s/22-Resource-Limits.html
Requesting resourceshttps://hamel.dev/notes/k8s/22a-Resource-Requests.html
Logginghttps://hamel.dev/notes/k8s/23-Logging.html
Monitoringhttps://hamel.dev/notes/k8s/24-monitoring.html
Ingresshttps://hamel.dev/notes/k8s/25-Ingress.html
Cluster Componentshttps://hamel.dev/notes/k8s/26-cluster.html
Securityhttps://hamel.dev/notes/k8s/security/index.html
Network Securityhttps://hamel.dev/notes/k8s/security/26-network-security.html
Securing Containershttps://hamel.dev/notes/k8s/security/27-container-security.html
Webhookshttps://hamel.dev/notes/k8s/security/28-workloads.html
Updating a K8s Clusterhttps://hamel.dev/notes/k8s/security/29-cluster-updates.html
RBAChttps://hamel.dev/notes/k8s/security/30-rbac.html
Workload Placementhttps://hamel.dev/notes/k8s/27-workload-placement.html
Auto Scalinghttps://hamel.dev/notes/k8s/28-auto-scaling.html
Preemptionhttps://hamel.dev/notes/k8s/29-preemption.html
Random TILshttps://hamel.dev/notes/k8s/99-Random.html
Linuxhttps://hamel.dev/notes/linux/index.html
Cheatsheethttps://hamel.dev/notes/linux/bash_scripting.html
Cookbookhttps://hamel.dev/notes/linux/cookbook.html
Misc Utilitieshttps://hamel.dev/notes/linux/misc_utils.html
OSX Shell Tipshttps://hamel.dev/notes/linux/osx.html
Processes, Permissions and Moving Datahttps://hamel.dev/notes/linux/permprocdata.html
LLMshttps://hamel.dev/notes/llm/index.html
AI Product Engineering Noteshttps://hamel.dev/notes/llm/ai-product-engineering/index.html
Automating Error Analysishttps://hamel.dev/notes/llm/ai-product-engineering/evals-error-analysis.html
Case Study: Putting Evals Into Productionhttps://hamel.dev/notes/llm/ai-product-engineering/evals-production.html
Evals for Data Agentshttps://hamel.dev/notes/llm/ai-product-engineering/evals-data-agents.html
Cut Classification Costs With a Model Cascadehttps://hamel.dev/notes/llm/ai-product-engineering/evals-model-cascades.html
An Intro to Multivector Retrievalhttps://hamel.dev/notes/llm/ai-product-engineering/context-multivector-retrieval.html
How to Improve Search Agentshttps://hamel.dev/notes/llm/ai-product-engineering/context-search-agents.html
How to Optimize Retrieval Embeddingshttps://hamel.dev/notes/llm/ai-product-engineering/context-embedding-models.html
How to Choose an OCR Modelhttps://hamel.dev/notes/llm/ai-product-engineering/context-ocr.html
Steer AI Writing With Footnotes for Agentshttps://hamel.dev/notes/llm/ai-product-engineering/context-subtext.html
Debugging Inference Latencyhttps://hamel.dev/notes/llm/ai-product-engineering/systems-inference-latency.html
Open Weight Model Economicshttps://hamel.dev/notes/llm/ai-product-engineering/systems-open-model-memory.html
The Case for Agent Sandboxeshttps://hamel.dev/notes/llm/ai-product-engineering/systems-agent-sandboxes.html
Turn Eval Results Into a Better Modelhttps://hamel.dev/notes/llm/ai-product-engineering/systems-post-training.html
Inferencehttps://hamel.dev/notes/llm/inference/index.html
Optimizing latencyhttps://hamel.dev/notes/llm/inference/inference.html
Max Inference Enginehttps://hamel.dev/notes/llm/inference/max_engine.html
vLLM & large modelshttps://hamel.dev/notes/llm/inference/big_inference.html
OpenAIhttps://hamel.dev/notes/llm/openai/index.html
Function promptshttps://hamel.dev/notes/llm/openai/func_template.html
AI evals resourceshttps://hamel.dev/notes/llm/evals/index.html
AI evals: Start herehttps://hamel.dev/notes/llm/evals/start/index.html
Evals: Getting startedhttps://hamel.dev/notes/llm/evals/start/new-to-evals/index.html
Evals: What to testhttps://hamel.dev/notes/llm/evals/start/getting-started/index.html
Evals: Can I trust my scores?https://hamel.dev/notes/llm/evals/start/trust-your-evals/index.html
Evals: Hard-to-evaluate productshttps://hamel.dev/notes/llm/evals/start/hard-to-evaluate/index.html
Evals: Reducing time and costhttps://hamel.dev/notes/llm/evals/start/reduce-eval-cost/index.html
Inspect AI, An OSS Python Library For LLM Evalshttps://hamel.dev/notes/llm/evals/inspect.html
Evals Flashcardshttps://hamel.dev/notes/llm/evals/flashcards/index.html
Evals Memeshttps://hamel.dev/notes/llm/evals/memes/index.html
Fine-tuninghttps://hamel.dev/notes/llm/finetuning/index.html
Dataset Basicshttps://hamel.dev/notes/llm/finetuning/datasets.html
LangChain DocumentLoadershttps://hamel.dev/notes/llm/finetuning/langchain_connectors.html
Estimating vRAMhttps://hamel.dev/notes/llm/finetuning/estimating_vram.html
Curating LLM datahttps://hamel.dev/notes/llm/finetuning/data_cleaning.html
Tokenization Gotchashttps://hamel.dev/notes/llm/finetuning/tokenizer_gotchas.html
Template-free axolotlhttps://hamel.dev/notes/llm/finetuning/template_free.html
Function Callinghttps://hamel.dev/notes/llm/tools/index.html
Llama-3 Func Callinghttps://hamel.dev/notes/llm/tools/llama3.html
RAGhttps://hamel.dev/notes/llm/rag/index.html
Stop Saying RAG Is Deadhttps://hamel.dev/notes/llm/rag/not_dead.html
P1: I don’t use RAG, I just retrieve documentshttps://hamel.dev/notes/llm/rag/p1-intro.html
P2: Modern IR Evals For RAGhttps://hamel.dev/notes/llm/rag/p2-evals.html
P3: Optimizing Retrieval with Reasoning Modelshttps://hamel.dev/notes/llm/rag/p3_reasoning.html
P4: Late Interaction Models For RAGhttps://hamel.dev/notes/llm/rag/p4_late_interaction.html
P5: RAG with Multiple Representationshttps://hamel.dev/notes/llm/rag/p5_map.html
P6: Context Rothttps://hamel.dev/notes/llm/rag/p6-context_rot.html
P7: You Don’t Need a Graph DB (Probably)https://hamel.dev/notes/llm/rag/p7-graph-db.html
Open Office Hourshttps://hamel.dev/notes/llm/officehours/index.html
Evals: Doing Error Analysis Before Writing Testshttps://hamel.dev/notes/llm/officehours/erroranalysis.html
Multi-Turn Chat Evalshttps://hamel.dev/notes/llm/officehours/evalmultiturn.html
Observability in LLM Applicationshttps://hamel.dev/notes/llm/officehours/observability.html
Tame Complexity By Scoping LLM Evalshttps://hamel.dev/notes/llm/officehours/scoping.html
How to Process Documents at Scale with LLMshttps://hamel.dev/notes/llm/data-processing/shreya-data-processing.html
Prompt engineeringhttps://hamel.dev/notes/prompt-eng/index.html
Coursehttps://hamel.dev/notes/prompt-eng/course/index.html
Guidelines for Promptinghttps://hamel.dev/notes/prompt-eng/course/01_guidelines.html
Iterative Prompt Developmenthttps://hamel.dev/notes/prompt-eng/course/02_iterative.html
Summarizinghttps://hamel.dev/notes/prompt-eng/course/03_summarizing.html
Inferringhttps://hamel.dev/notes/prompt-eng/course/04_inferring.html
Transforminghttps://hamel.dev/notes/prompt-eng/course/05_transforming.html
Expandinghttps://hamel.dev/notes/prompt-eng/course/06_expanding.html
The Chat Formathttps://hamel.dev/notes/prompt-eng/course/07_chatbot.html
Quartohttps://hamel.dev/notes/quarto/index.html
Syntax Highlightinghttps://hamel.dev/notes/quarto/highlighting.html
Listings from datahttps://hamel.dev/notes/quarto/listings-from-data.html
Merge listingshttps://hamel.dev/notes/quarto/merging.html
ML Servinghttps://hamel.dev/notes/serving/index.html
TF Servinghttps://hamel.dev/notes/serving/tfserving/index.html
Basicshttps://hamel.dev/notes/serving/tfserving/tf-serving-basics.html
GPUs & Batchinghttps://hamel.dev/notes/serving/tfserving/gpu.html
TorchServehttps://hamel.dev/notes/serving/torchserve/index.html
Basicshttps://hamel.dev/notes/serving/torchserve/basic-torchserve.html
Serving Your Own Modelhttps://hamel.dev/notes/serving/torchserve/hf.html
FastAPIhttps://hamel.dev/notes/serving/fastapi/index.html
Web Scrapinghttps://hamel.dev/notes/web-scraping/index.html
Browser requests to codehttps://hamel.dev/notes/web-scraping/browser-to-python.html
Transcribe & Diarize Videoshttps://hamel.dev/notes/web-scraping/transcribe-diarize.html
Videohttps://hamel.dev/notes/llm/evals/inspect.html#video
Annotated Presentationhttps://hamel.dev/notes/llm/evals/inspect.html#annotated-presentation
1. Inspect Overviewhttps://hamel.dev/notes/llm/evals/inspect.html#inspect-overview
2. Motivationhttps://hamel.dev/notes/llm/evals/inspect.html#motivation
3. Core Conceptshttps://hamel.dev/notes/llm/evals/inspect.html#core-concepts
4. Hello, World Examplehttps://hamel.dev/notes/llm/evals/inspect.html#hello-world-example
5. Two Views of Inspecthttps://hamel.dev/notes/llm/evals/inspect.html#two-views-of-inspect
6. High Level API Exampleshttps://hamel.dev/notes/llm/evals/inspect.html#high-level-api-examples
7. Low Level APIhttps://hamel.dev/notes/llm/evals/inspect.html#low-level-api
8. Solvershttps://hamel.dev/notes/llm/evals/inspect.html#solvers
9. Baseline Solvershttps://hamel.dev/notes/llm/evals/inspect.html#baseline-solvers
10. Multiple Choice Solverhttps://hamel.dev/notes/llm/evals/inspect.html#multiple-choice-solver
11. Self Critique Solverhttps://hamel.dev/notes/llm/evals/inspect.html#self-critique-solver
12. Compositionhttps://hamel.dev/notes/llm/evals/inspect.html#composition
13. Composition Examplehttps://hamel.dev/notes/llm/evals/inspect.html#composition-example
14. Tool Usehttps://hamel.dev/notes/llm/evals/inspect.html#tool-use
15. Agents and Toolshttps://hamel.dev/notes/llm/evals/inspect.html#agents-and-tools
16. Agent APIhttps://hamel.dev/notes/llm/evals/inspect.html#agent-api
17. Agent Toolshttps://hamel.dev/notes/llm/evals/inspect.html#agent-tools
18. Agent Approvalhttps://hamel.dev/notes/llm/evals/inspect.html#agent-approval
19. Agent Evalshttps://hamel.dev/notes/llm/evals/inspect.html#agent-evals
20. Scoringhttps://hamel.dev/notes/llm/evals/inspect.html#scoring
21. Custom Scorer Examplehttps://hamel.dev/notes/llm/evals/inspect.html#custom-scorer-example
22. Production: Parallelismhttps://hamel.dev/notes/llm/evals/inspect.html#production-parallelism
23. Production: Eval Setshttps://hamel.dev/notes/llm/evals/inspect.html#production-eval-sets
24. Production: Logginghttps://hamel.dev/notes/llm/evals/inspect.html#production-logging
25. EvalLog Structurehttps://hamel.dev/notes/llm/evals/inspect.html#evallog-structure
26. Log Viewer Screenshothttps://hamel.dev/notes/llm/evals/inspect.html#log-viewer-screenshot
27. Sample Detail Viewhttps://hamel.dev/notes/llm/evals/inspect.html#sample-detail-view
28. Message History Viewhttps://hamel.dev/notes/llm/evals/inspect.html#message-history-view
29. Assistant Thinking Viewhttps://hamel.dev/notes/llm/evals/inspect.html#assistant-thinking-view
30. Inspect View IDEhttps://hamel.dev/notes/llm/evals/inspect.html#inspect-view-ide
32. Sandbox Environmentshttps://hamel.dev/notes/llm/evals/inspect.html#sandbox-environments
33. Tool Environment Typeshttps://hamel.dev/notes/llm/evals/inspect.html#tool-environment-types
34. Learning Morehttps://hamel.dev/notes/llm/evals/inspect.html#learning-more
Q&A Sessionhttps://hamel.dev/notes/llm/evals/inspect.html#qa-session
Noteshttps://hamel.dev/notes/index.html
LLMshttps://hamel.dev/notes/llm/index.html
AI evals resourceshttps://hamel.dev/notes/llm/evals/index.html
Inspect AI, An OSS Python Library For LLM Evalshttps://hamel.dev/notes/llm/evals/inspect.html
JJ Allairehttps://en.wikipedia.org/wiki/Joseph_J._Allaire
LLM Evals coursehttps://maven.com/parlance-labs/evals?promoCode=evals-info-book
Posithttps://posit.co/
Inspect AIhttps://inspect.ai-safety-institute.org.uk/
annotated presentationshttps://simonwillison.net/tags/annotated-talks/
AI Evals coursehttps://maven.com/parlance-labs/evals?promoCode=evals-info-book
Timestamp: 01:50https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=110s
Timestamp: 01:23https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=83s
Timestamp: 03:52https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=172s
Timestamp: 04:44https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=284s
Timestamp: 07:02https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=422s
Timestamp: 08:06https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=486s
Timestamp: 09:47https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=587s
Timestamp: 11:27https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=687s
Timestamp: 11:52https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=712s
Timestamp: 12:24https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=744s
Timestamp: 14:02https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=842s
Timestamp: 14:34https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=874s
Timestamp: 15:03https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=903s
Timestamp: 15:56https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=956s
bridgeshttps://inspect.aisi.org.uk/agent-bridge.html
Timestamp: 16:58https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1018s
Timestamp: 20:00https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1200s
Timestamp: 21:09https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1269s
browser use toolhttps://inspect.aisi.org.uk/tools-standard.html#sec-web-browser
Timestamp: 21:52https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1312s
Timestamp: 23:53https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1433s
Timestamp: 24:08https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1448s
Timestamp: 24:32https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1472s
Timestamp: 24:45https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1485s
Timestamp: 24:49https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1489s
Timestamp: 24:51https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1491s
Timestamp: 24:54https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1494s
Timestamp: 25:00https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1500s
Timestamp: 25:02https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1502s
Timestamp: 25:05https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1505s
Timestamp: 25:12https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1512s
Timestamp: 26:04https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1603s
Timestamp: 26:54https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1614s
Timestamp: 28:13https://youtu.be/_UY49Q_qFhs?si=t9-YfrS4JN_UAbpM&t=1693s
AI Evals coursehttps://maven.com/parlance-labs/evals?promoCode=evals-info-book
https://www.linkedin.com/in/hamelhusain/
https://twitter.com/HamelHusain
https://github.com/hamelsmu

Viewport: width=device-width, initial-scale=1.0, user-scalable=yes


URLs of crawlers that visited me.