Agentic Behavior Observatory

Definitions

The terms this observatory measures, in the sense it measures them. Where a term is contested, the reading used here is stated plainly rather than smoothed over — a score means nothing unless you know what it was counting.

The behavior itself

Reinforcement behavior

Behavior that is learned from consequence rather than specified in advance. An agent acts, the environment returns a reward, and the agent adjusts what it will do next so as to earn more reward over time. Nobody writes down the strategy; the strategy is what survives repeated feedback.

The distinction that matters for this observatory: rule-based behavior is authored — you can read the rule in the source and know what the agent will do. Reinforcement behavior is discovered — you can read the reward function and still not know what the agent will do, which is exactly why it has to be observed rather than assumed. Reward hacking, where an agent maximizes the stated reward while defeating its intent, is the standard demonstration of that gap.

Signals: named algorithms (PPO, DQN, SAC, Q-learning), reward functions and discount factors, the reset/step environment contract, RLHF and DPO.

Reinforcement learning (RL)

The family of methods that produce reinforcement behavior: an agent, an environment, a state, a set of actions, a reward signal, and a policy mapping states to actions. Training searches for a policy that maximizes expected cumulative reward, discounted over time.

Its relevance here is twofold. It is how many simulated agents acquire their behavior — and, through RLHF and preference optimization, it is how the language models used as agents acquired theirs. A model's behavioral tendencies in a simulation are partly an artifact of the reward model used to align it, which is one reason model version matters as much as prompt design.

Agentic behavior

Behavior of a system that pursues goals over multiple steps, keeps state between them, chooses among available actions, and acts on an environment rather than only answering. The word marks a spectrum, not a category: a scripted chatbot sits at one end, a tool-using planner that revises its approach after failure sits at the other.

This observatory treats agency as an empirical property of the code — does something loop, hold memory, select actions, receive consequences? — rather than a claim in the README.

Emergence

Population-level regularities that no individual agent's rules state: segregation from mild preferences, traffic waves from local braking, opinion cascades from local imitation. It is the usual reason for simulating a population instead of reasoning about the average member of it.

It is also the usual place validation fails. An emergent pattern can be a genuine finding or an artifact of the scheduler, the grid, or the random seed, which is why seeded runs and sensitivity analysis carry weight on the evaluation axis.

The three subject axes

Agent-based simulation (ABM)

Modeling a system as many interacting individuals, each following its own rules, and running them forward in time to see what the population does. The unit of description is the individual; aggregate behavior is an output, never an input.

Signals: Mesa, AgentPy, PettingZoo, NetLogo, Repast, SimPy; agent classes, schedulers, step loops, spatial grids and networks.

Synthetic data & synthetic populations

Synthetic data is generated rather than observed, built to preserve the statistical structure of real data without carrying its records. A synthetic population is the specific case that matters here: a full roster of individuals with plausible, jointly consistent attributes — age with income with household with region — matching known marginals from a census or survey, so a simulation can be run over people who do not exist but who are distributed like people who do.

The classic construction is iterative proportional fitting, reweighting a seed sample until it matches published margins. The classic failure is a population that matches every margin and no real joint distribution — every attribute right one at a time, nobody plausible taken whole.

Signals: SDV, CTGAN, synthcity, Faker, Gretel; population synthesis and IPF, census/microdata seeds, differential privacy, generation at scale.

LLM-based behavioral modeling

Using a language model as the behavioral engine of a simulated individual: the agent is given a persona, a memory, and a situation, and the model's output is its behavior. It replaces hand-written decision rules with a model's learned priors about how someone so described would act.

The trade is explicit. Hand-written rules are transparent and thin; model-driven agents are rich and opaque, carrying whatever the training data encoded about the demographic they are asked to portray — which is why the dimensions a study conditions on, and the model version it used, are both recorded here.

Signals: Anthropic/OpenAI SDKs, LangGraph, AutoGen, CrewAI, DSPy, local runtimes; personas, memory streams, generative-agent architectures.

The method axes

These describe how work is done rather than what it is about, so they shape a repository's profile without entering the headline relevance. Reinforcement learning is the third; it is defined above, beside the behavior it produces.

Evaluation & validation

The apparatus that decides whether a simulation's output means anything: tests, fidelity metrics against real distributions, sensitivity analysis and ablations, seeded runs that reproduce, experiment tracking, and audits of bias and representativeness.

It scores as a method axis rather than a subject axis because it says how the work is checked, not what it is about. Read it beside the subject scores, not instead of them: a repository can score high here for having a thorough test suite around a thin model, and research code released to accompany a published paper can score zero because the validation lived in the paper.

Signals: test suites and CI, KS / Wasserstein / TSTR, sensitivity analysis and ablations, seeded runs, experiment tracking, bias and representativeness audits.

Context isolation (and context bleeding)

Context bleeding is information reaching a place the study assumes it cannot: one agent reading another's private brief, a run inheriting the previous run's memory, a model answering out of its own training rather than the persona it was given, or an evaluation item sitting in the prompt that is meant to test it. It is a quiet failure, because the output still looks like a result.

The axis scores the guards, not the leak, exactly like every other axis here: a high score means the repository has the vocabulary of keeping contexts apart, not that it is bleeding. It is a method axis, so it shapes the profile without touching the headline relevance.

Read a low score as a boundary nobody wrote down, not as proof of a leak. Silence is the absence of evidence in both directions, which is the whole reason each signal links to the line it fired on.

Signals: private and privileged instructions, BATNAs and red lines; visibility and disclosure rules; per-agent scoped context; fresh sessions and state resets; independent replications; staying in character and knowledge-cutoff handling; contamination and leakage checks; blind evaluation.

Terms of art

Silicon sampling

Prompting a language model to answer as a demographically specified person, and treating the answers as survey responses. Cheap, fast, and reproducible; also prone to reproducing a flattened, stereotyped version of the group being portrayed, and to shifting when the model version changes underneath the study.

Generative agents

The architecture pairing a language model with a memory stream, retrieval over that memory, periodic reflection into higher-level beliefs, and planning — so behavior accumulates a history instead of restarting each turn.

Persona conditioning

The attributes supplied to an agent before it acts — age, occupation, region, beliefs, backstory. This is where a study's demographic assumptions actually live, usually in a prompt template rather than in a documented model, which is why the observatory extracts them.

Population dimension

A single attribute along which a synthetic population varies — age, income, education, region, migration status, disability, literacy, caste. The observatory counts which dimensions appear across the corpus, because a dimension nobody models is a population nobody simulates, and an unmodeled dimension is an implicit assumption that it does not matter.

Fidelity, utility, privacy

The three axes any synthetic population is judged on. Fidelity: does it resemble the real distribution (KS, Wasserstein, Jensen–Shannon)? Utility: does a model trained on it work on real data (train-synthetic-test-real)? Privacy: can a real individual be recovered from it (differential privacy, k-anonymity, re-identification tests)? Gains on any one are usually paid for by another.

Face validity vs. construct validity

Face validity: the simulation looks right to someone who knows the domain. Construct validity: the mechanism inside it corresponds to the mechanism in the world. Model-driven agents make the first easy to achieve and the second no easier — a plausible transcript is not evidence of a correct mechanism.

Building synthetic populations

A practical guide to constructing a synthetic population, the decisions that shape it, and how each of those decisions propagates into the behavior of the agents built on top of it. Written for the case this observatory tracks: populations that will be acted through by simulated agents, not merely analyzed as a table.

The through-line: a synthetic population is a set of assumptions wearing the costume of data. Every step below converts a judgment into a number that later looks like a fact. The work is keeping the judgments visible.

Step 0 — Decide what the population is for

The intended decision determines which parts of the population must be right, and there is no such thing as a generally correct one. A population fit for estimating total evacuation time is not fit for estimating who gets left behind, because the second question lives entirely in the tail the first one averages away.

Write the decision down before generating anything, in this form: "This population is fit for estimating ___ for ___ under ___, and is not fit for ___." Everything downstream is a trade-off, and without that sentence you cannot say which side of each trade-off to take.

The pipeline

1. 1 · Unit and closure

   Choose the unit of the model — individual, household, firm, vehicle, school — and be aware that behavior usually happens at a different level from the data. Evacuation decisions are made by households; census attributes describe individuals. Whichever you pick, the other has to be constructed, and that construction is a modeling assumption, not a lookup.

   Closure is the harder half: who counts as in the population? Residents, or people present? A daytime commuter population differs from a nighttime residential one by a factor that dwarfs most parameter uncertainty. Populations that are unregistered, displaced, undocumented, institutionalized, or unhoused are missing from nearly every seed and every margin — and in humanitarian work they are frequently the population of interest. Their absence is not neutral; it silently sets their count to zero.

2. 2 · Seed and margins

   The standard construction needs two ingredients: a seed — a sample of real individuals or households with the joint structure of their attributes intact (census microdata, PUMS, IPUMS, DHS, a household survey) — and margins — published totals for the area you are populating (counts by age, by tenure, by household size).

   The seed supplies the joints; the margins supply the totals. Both carry their own history: a seed drawn from a different region imports that region's correlations, and a seed drawn a decade ago imports a decade-old society. Record the vintage and the geography of both, because every later claim inherits them.

3. 3 · Fit

   Reweight or resample the seed until it reproduces the margins. Methods, honestly compared:

   Method	Strength	Cost
   IPF (iterative proportional fitting)	Simple, fast, reproducible, well understood	Fractional weights; cannot invent combinations absent from the seed
   IPU / hierarchical fitting	Fits individual and household margins at once	More fragile; convergence is not guaranteed
   Combinatorial optimization	Whole individuals, no fractional weights	Expensive; can overfit the margins exactly
   Copulas / CTGAN / VAE	Learns joints, generates unseen but plausible combinations	Opaque; can hallucinate impossible people; needs its own validation
   LLM generation	Rich, unstructured attributes (occupation, narrative, preference)	No distributional guarantee whatsoever; imports training-data priors as if they were population statistics

   The dominant failure is the same across all of them: matching every margin while matching no real joint distribution. Each attribute is right one at a time, and nobody in the population is a plausible whole person. Fit is checked on margins; validity lives in the joints.

4. 4 · Structure — households, space, networks

   Individuals alone are not a population. Three structures have to be constructed, and each one usually matters more to the simulation's output than the attribute distributions everyone spends their time on:

   • Household composition — who lives with whom. Constrained by logic (a five-year-old is not a household head) and by joint distributions that margins never publish. Household structure governs who evacuates together, whose mobility limits whose, and who cannot leave because of someone else.
   • Spatial allocation — where each agent is placed. Assign at too coarse a resolution and you erase the local heterogeneity that drives the result; assign at too fine a resolution and you invent precision the source data cannot support. This is the modifiable areal unit problem: the same underlying reality yields different findings under different zone boundaries.
   • Social network — who talks to whom. Almost never in the source data, almost always decisive. If your model has any diffusion in it — information, panic, imitation, disease — the network topology and its homophily will drive the outcome harder than any individual attribute. A network assumed uniformly random is a strong claim that a society is unsegregated.

5. 5 · Attach behavior — the load-bearing step

   Here demographics become conduct: the mapping from an agent's attributes to its behavioral parameters — walking speed, risk tolerance, trust in official warnings, time to comply, propensity to wait for a relative.

   This mapping is where most of a study's real content lives, and it is the least documented part of nearly every model. Two teams with the identical population and opposite conclusions almost always differ here, not in the fitting. Three rules follow:

   • Every attribute→parameter link needs a citation, an assumption label, or a calibration target. "Older agents move slower" needs a distribution and a source, not a coefficient someone liked.
   • Prefer mappings to situations over mappings to identities. "Households with a dependent adult wait longer" is a mechanism; "elderly people are non-compliant" is a stereotype wearing a parameter's clothes, and it will reproduce itself as a finding.
   • Keep the mapping in one inspectable place — a table, not scattered literals. If a reviewer cannot list your behavioral assumptions in five minutes, neither can you.

6. 6 · Validate on three axes

   A synthetic population is judged on fidelity, utility, and privacy, and gains on one are usually paid for by another.

   • Fidelity — does it resemble the real distribution? Compare margins (trivially true by construction, so nearly worthless as evidence) and then the held-out joints the fit never targeted (the actual test). KS, Wasserstein, Jensen–Shannon per attribute; cross-tabulations for pairs.
   • Utility — does it work for the task? Train-synthetic-test-real, or re-run a known historical case and compare to what happened. Utility for your decision, not utility in general.
   • Privacy — can a real person be recovered? Especially where the population is small and an attribute combination is rare. Differential privacy and k-anonymity both work by damaging exactly the rare cells that humanitarian analysis cares most about; that collision is a design decision to state, not a box to tick.

   Add face validity — does it look right to someone who knows the place? — but never mistake it for construct validity. A plausible-looking population can be built on a wrong mechanism, and a model-driven one is especially good at looking plausible.

7. 7 · Generate many, not one

   A single population is false precision. Margins carry sampling error, the fit has stochastic choices, and the network was assumed. Generate an ensemble across seeds and report the spread of your conclusion, not the mean of one run.

   The useful question is not "what does the model say?" but "which conclusions survive a different plausible population?" Those are your findings. The rest is one draw from a distribution you did not look at.

8. 8 · Document it as data, not as code

   Ship a datasheet with the population: seed source and vintage, margins used, geography and unit, fitting method and convergence, what was assumed rather than fitted (networks, households, behavior parameters), known missing groups, privacy treatment, validation results, and the fitness statement from Step 0. Without it the population will be reused for a question it was never fit for — which is the most common way this work goes wrong, and it usually happens to a colleague six months later.

The complexities, stated plainly

Marginal fit is not joint fit

Every attribute correct in aggregate, no plausible individuals. The canonical failure, and invisible to every margin-based check.

Structural vs. sampled zeros

A combination absent from your seed may be impossible (a toddler with a doctorate) or merely unobserved (a rare but real household type). Treating the second as the first erases real people; treating the first as the second invents impossible ones.

Rare subgroups carry the result

Small groups are exactly where fitting error, privacy noise, and seed gaps concentrate — and in evacuation, health, or exclusion work they are usually the population the study exists to serve. Precision is worst precisely where it matters most.

The ecological fallacy and the MAUP

Area-level rates are not individual probabilities, and conclusions can change when zone boundaries change. Both convert a data-resolution artifact into a substantive finding.

Temporal drift

Censuses are old; conflict, displacement, and migration move faster than statistical agencies. A 2019 seed describes a society that may no longer exist in a 2026 crisis.

Categories are political

Race, gender, caste, ethnicity, and legal status schemas encode a particular state's way of seeing its people. Adopting them silently adopts that view — including its omissions, which are rarely accidental.

Missing populations

Undocumented, displaced, unregistered, unhoused, and institutionalized people are absent from most seeds and margins. Their absence is recorded as a zero, and a zero is a claim.

Assumed structure dominates fitted structure

The network you assumed and the behavior mapping you wrote will usually drive the output more than the distributions you carefully fitted. Effort tends to be allocated in inverse proportion to influence.

Privacy noise lands on the vulnerable

Differential privacy protects rare combinations by distorting them — the same cells that describe minority and at-risk groups. Protection and representation are in genuine tension here.

Validation circularity

Validating against the margins you fitted proves only that the fitting worked. Real validation needs a held-out joint, an external dataset, or a known outcome.

How population choices become agentic behavior

Each link in this chain converts a decision into an apparent fact. Distortion introduced early is amplified, not averaged out, because emergent outcomes are driven by tails and structure rather than means.

Three consequences deserve naming, because they are specific to agentic models and are not solved by better fitting:

• The tail drives the emergent result. Aggregate-accurate populations routinely produce wrong dynamics, because cascades, bottlenecks, and failures-to-evacuate are generated by the least typical agents. A population validated on means is not validated for a model whose output depends on extremes.
• LLM agents fill the gaps you left, from training priors. Whatever the population underspecifies, a language model supplies — confidently, and from patterns in its training data about how someone so described behaves. Two documented effects follow: flattening, where a demographic is portrayed as more internally uniform than any real group, and stereotype amplification, where the modal association becomes the modeled behavior. This converts an unstated assumption into what looks like a finding about a real group.
• Model version is a study parameter. A model-driven population's behavior shifts when the underlying model changes, since alignment training reshapes exactly the dispositions being simulated. A result reproducible only against a deprecated checkpoint is a result with a shelf life — which is why this observatory records the model versions each repository rests on.

Pre-flight checklist

1. Can you state what this population is fit for, and what it is not fit for?
2. Do you know the vintage and geography of your seed, and whose society it describes?
3. Which groups are missing from the seed and margins, and does the study's question concern them?
4. Have you validated a joint distribution the fit never targeted?
5. Is the household, spatial, and network structure assumed or derived — and do you know which?
6. Is every attribute→parameter mapping in one inspectable place, each with a source or an explicit assumption label?
7. Are those mappings keyed to situations rather than identities?
8. Do you report a spread across replicate populations, or a single run?
9. Does the conclusion survive a plausible alternative population?
10. If a language model supplies behavior: which version, with what prompt, and what does it invent that you did not specify?
11. Could a real individual be re-identified, and what did the privacy treatment cost your rare subgroups?
12. Is there a datasheet a stranger could read before reusing this?

What the observatory can see of this

The scores on the Observatory tab detect the machinery of good practice — a fitting method, seeded runs, fidelity metrics, privacy treatment, a sensitivity analysis, the demographic dimensions a repository conditions on. That is real evidence, and it is why the evaluation axis exists.

It cannot see whether a mapping from age to compliance was defensible, whether the missing population was the one that mattered, or whether the network was assumed into the shape that produced the finding. Those are the questions on this page, and they are answered by reading the code and asking the authors — the score only tells you where to look.

How this is built

Paste a GitHub URL and the analysis runs in your browser — there is no server. What follows is the whole method, so you can judge a score rather than trust it.

1. 1 · Read the repository

   Two calls to api.github.com fetch the repository metadata and its full file tree. Every file body then comes from raw.githubusercontent.com, which is CORS-open and not rate limited — so a complete analysis costs two API requests, roughly thirty repositories an hour with no token at all.

   Vendored and build directories (node_modules, dist, venv, …) are dropped. Of what remains, up to 120 text files under 200 KB are read, ordered: dependency manifests first, then prose, then source — and within source, paths whose names suggest the modeling core (agent, persona, popul, synth, reward, prompt…) before the rest, larger files before near-empty ones. Every report states files_read / files_total, so the sampling is never silent.

2. 2 · Fire signals

   The taxonomy is a flat list of signals, each belonging to one axis and carrying a weight. A signal fires on the first match of its pattern against one of three surfaces:

   • dep — the names declared in requirements.txt, pyproject.toml, package.json, and friends
   • path — the repository's file paths
   • text — the contents of the files that were read

   A fired signal records where it fired, down to the line, plus the surrounding text. That evidence is attached to the report and shown in the dashboard, so any score can be walked back to the code that produced it.

3. 3 · Score the axes

   Each axis scores as the share of its available weight that fired:

   axis score = 100 × (weight of signals that fired) / (total weight on that axis)

   This is signal coverage, not quality. A framework that touches everything scores above a small, sharp research repository that does one thing exactly right — and that is the intended reading, not a flaw to correct for.

4. 4 · Set the headline

   Three axes describe the subject of the work — agent-based simulation, synthetic data, LLM behavioral modeling. Two describe how it is done — reinforcement learning and evaluation. The headline relevance uses only the subject axes:

   relevance = 0.6 × max(subject axes) + 0.4 × mean(subject axes)

   The max term keeps a purely agent-based or purely synthetic-data repository squarely in scope; the mean term rewards work that genuinely spans them. The primary axis shown on each card is the strongest subject axis, never the evaluation score.

5. 5 · Extract the population lab

   Two things get pulled out of the text separately from scoring, because they are the actual research question:

   Dimensions modeled. A fixed vocabulary of demographic attributes — age, income, education, region, migration, disability, caste, literacy, and so on — is matched against every file read, and the number of files mentioning each is recorded. Across the corpus this shows which population dimensions the field models and which it never touches. A dimension nobody models is a population nobody simulates.

   Models and versions. Every model identifier (claude-*, gpt-*, gemini-*, llama-*, and others) is counted. Behavioral findings drift between model versions, and the version a study rests on is usually buried in a config file. Here it is on the card.

One taxonomy, two runtimes

The scoring lives in analyzer/signals.py and is exported to docs/data/signals.json, which the browser analyzer loads. The Python CLI and this page therefore score identically — the dashboard is not an approximation of the real thing.

Teaching it a new framework is one line:

("abm", "vadere", "dep", r"^vadere$", 10, "Vadere crowd simulation")
#  axis   key      kind   pattern      weight  label

Run analyzer/build_index.py and both runtimes pick it up.

Browser runs versus committed runs

An analysis you run here is yours: it appears in the grid marked live and is kept in this browser. It becomes part of the shared corpus only when its report is committed to data/reports/ — download the JSON from the detail panel, or run the CLI:

python analyzer/analyze.py https://github.com/owner/repo

That writes the JSON and a Markdown brief, and rebuilds the index the dashboard reads.

What this cannot tell you

• Regular expressions match vocabulary, not meaning. A repository that discusses differential privacy without implementing it fires that signal. Evidence links exist precisely so you can check.
• Large repositories are sampled, not read whole. A signal can be missed because its file ranked 121st.
• Scores compare a repository against the taxonomy — never against another repository's scientific merit.
• The demographic vocabulary is a fixed English list. It will under-report populations described in other terms, and that limit is itself a finding worth reading.