> ## Documentation Index
> Fetch the complete documentation index at: https://docs.expectedparrot.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Dataset

> Work with selected columns as tabular data and export them for analysis.

A `Dataset` is a table of columns. A `Results` object retains interview and study information; calling `results.select(...)` produces a Dataset containing the selected columns. Use Dataset methods or pandas to transform and export that table.

For example, select a treatment label and a rating from Results, filter to the observations you want, then export a CSV for your statistical software. You can do these steps directly in EDSL; converting with `.to_pandas()` is useful when you want pandas analyses or plotting. A [ScenarioList](https://docs.expectedparrot.com/en/latest/scenarios.md) is the corresponding collection of named records used as inputs to questions.

## A complete local example

This example uses four fabricated scores and makes no model calls. It creates a Dataset from a DataFrame, filters and sorts rows, adds a derived column, and writes a CSV.

```python theme={null}
import pandas as pd
from edsl import Dataset

frame = pd.DataFrame({
    "condition": ["short", "explicit", "short", "explicit"],
    "score": [3, 5, 2, 4],
})
dataset = Dataset.from_pandas_dataframe(frame)
selected = dataset.filter("score >= 4").order_by("score", reverse=True)
with_flag = dataset.mutate("high_score = score >= 4")
# Display in a notebook: put selected on the last line of a cell.
# Display in a script: selected.print()
# Convert to pandas when you need DataFrame operations.
analysis_frame = selected.to_pandas()
selected.to_csv("selected_scores.csv")
```

`analysis_frame` contains the two `explicit` rows with scores 5 and 4, in that order. `with_flag` contains all four rows plus the Boolean `high_score` column. `selected_scores.csv` contains only the selected rows.

## Constructors and conversions

Use `Dataset.from_pandas_dataframe(frame)` for a pandas DataFrame. You can also construct a column-oriented table directly:

```python theme={null}
from edsl import Dataset

dataset = Dataset([{"record_id": ["r1", "r2"]}, {"score": [3, 5]}])
frame = dataset.to_pandas()
```

Keep columns the same length. `to_pandas()` preserves prefixed EDSL column names by default. Use qualified names when joining data so unrelated columns are not confused.

## Common methods

| Call | Arguments and result |
| - | - |
| `dataset.select("condition", "score")` | Return a Dataset with the selected columns. |
| `dataset.filter("score >= 4")` | Return rows matching a Boolean expression. |
| `dataset.order_by("score", reverse=True)` | Return rows sorted by the named column; descending when `reverse=True`. |
| `dataset.mutate("high_score = score >= 4")` | Return a Dataset with a new or updated column computed from the expression. |
| `dataset.to_list()` | Return a list of values for one column, or a list of row tuples for multiple columns. |
| `dataset.to_pandas(remove_prefix=False)` | Return a pandas DataFrame; set `remove_prefix=True` to strip namespaces such as `answer.`. |
| `dataset.to_csv("scores.csv")` | Write a CSV file and return a FileStore containing the CSV. Column prefixes are retained by default. |
| `dataset.print()` | Display the table explicitly in a script. A final notebook expression displays it automatically. |

This example filters a table and converts a single selected column to a Python list:

```python theme={null}
from edsl import Dataset

scores = Dataset([{"score": [3, 5, 2, 4]}])
high_scores = scores.filter("score >= 4").order_by("score", reverse=True)
high_scores
# In a script: high_scores.print()
# For further processing: high_scores.select("score").to_list() returns [5, 4].
```

Conversion is optional when you only want to inspect the table. See [Results](https://docs.expectedparrot.com/en/latest/results.md) for selecting columns from a completed study.

## Reshape and inspect a table

| Task | Methods and behavior |
| - | - |
| Preview rows | `head(n=5)`, `tail(n=5)`, or `table(max_rows=5)` returns a smaller table. `first()` returns the first value in the first column. |
| Inspect structure | `keys()` lists column names; `summary()` describes the table. |
| Rename or remove columns | `rename({"old": "new"})`, `drop("column")`, and `remove_prefix()` return new Datasets with renamed, removed, or unprefixed columns. Check for collisions before removing prefixes. |
| Sample rows | `sample(n=2, with_replacement=False, seed=42)` returns a sample; `shuffle(seed=42)` returns a shuffled table. Dataset sampling defaults to replacement, so choose explicitly. |
| Deduplicate | `unique()` returns distinct rows. |
| Reshape lists | `expand("tags")` creates rows from list items; `collapse("text", separator="\n")` combines values while grouping by the other columns. |
| Reshape all columns | `long()` produces `row`, `key`, and `value` columns; `wide()` reconstructs that layout. |
| Join tables | `merge(other, by_x="record_id", by_y="id")` performs a left join. Check for duplicate keys first. |
| Format output | `to_markdown_table()` returns a Markdown table string; `latex()` returns a LaTeX table. `print()` explicitly displays the table in scripts. |

For example, inspect the long representation of a small table:

```python theme={null}
from edsl import Dataset

table = Dataset([{"record_id": ["r1", "r2"]}, {"score": [3, 5]}])
long_table = table.long()
long_table
# long_table.wide() reconstructs the original columns.
```

## Export and convert tables

These methods also apply to Results, ScenarioList, and AgentList. Select the fields you want before exporting:

| Output | Call |
| - | - |
| Row dictionaries | `table.to_dicts(remove_prefix=False)` |
| Python values | `table.select("score").to_list()` returns the values of one column. Multiple columns produce row tuples. |
| pandas or Polars | `table.to_pandas()` / `table.to_polars()`; prefixes are retained by default. |
| CSV | `table.to_csv("scores.csv")` |
| Excel | `table.to_excel("scores.xlsx", sheet_name="Scores")` |
| Word table | `table.to_docx("scores.docx")` |
| EDSL inputs | `table.to_scenario_list()` / `table.to_agent_list()`; these remove column prefixes by default. |

The CSV, Excel, and Word export methods above return `FileStore` objects. `table.write("scores.csv")` writes a file directly, choosing its format from the filename extension.

`table.to_json()` returns a Python list of column dictionaries. Use `json.dumps()` when you need JSON text:

```python theme={null}
import json

json.dumps(table.to_json())
```

This differs from `table.to_dict()`, which returns the EDSL object structure that `Dataset.from_dict()` can reconstruct.

Install the corresponding optional library for conversions such as `to_polars()` or Excel output. For database exports, convert to a pandas DataFrame and use its database-writing methods with your chosen connection and replacement policy.

## Embed text

Text embeddings represent passages as lists of numbers that can be used to search for similar passages or group related responses. `EmbeddingModel` creates these representations; `Dataset.embed()` and `ScenarioList.embed()` add them as a column beside the source text. Embedding the text does not itself assign themes or create groups. Use [qualitative coding](https://docs.expectedparrot.com/en/latest/research/qualitative.md) when you want named themes from a codebook.

This example uses fixed test vectors to show the table structure without a provider call. Their values do not represent semantic similarity:

```python theme={null}
from edsl import Dataset, EmbeddingCache, EmbeddingModel

texts = Dataset([{"text": ["The tool is easy to use.", "Exporting takes too long."]}])
embedding_model = EmbeddingModel(service_name="test", dimensions=4)
embedding_cache = EmbeddingCache()
embedded = texts.embed("text", model=embedding_model, cache=embedding_cache)
embedded
```

For actual text embeddings, select an embedding model and service explicitly: `EmbeddingModel(model_name="EMBEDDING_MODEL_NAME", service_name="openai")` uses direct provider access by default. Configure the provider's credentials. The `sentence_transformers` service runs locally with its dependencies and model weights installed; loading a model may download weights. `remote=True` routes supported embedding requests through Expected Parrot. Embedding models are separate from the language models selected with `Model()`.

For text outside a Dataset, `embedding_model.embed(["first text", "second text"], cache=embedding_cache)` returns an `EmbeddingResult`. Its `embeddings`, `input`, `model`, `service_name`, `usage`, and `cache_used` attributes retain the vectors and available metadata. `to_dataset()` and `to_scenario_list()` convert it to EDSL tables; `to_dict()` and `EmbeddingResult.from_dict(data)` serialize and restore it. In asynchronous Python, use `await embedding_model.async_embed(texts, cache=embedding_cache)`.

`EmbeddingCache(filename="embeddings.jsonl")` loads existing embedding entries and writes new ones to that file. With `immediate_write=False`, call `flush()` to persist pending entries. This cache is separate from the response Cache used by survey runs. Embeddings and provider calls can incur costs; the test service above makes no paid calls.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.