Generating synthetic data¶
Marvin can generate synthetic data according to a schema and instructions. Generating synthetic data with an LLM can yield extremely rich and realistic samples, making this an especially useful tool for testing code, training or evaluating models, or populating databases.
What it does
generate function creates synthetic data according to a specified schema and instructions.
We can generate a variety of names by providing instructions. Note the default behavior is to generate a list of strings:
By providing a target type, we can generate dictionaries that map countries to their populations:
Pydantic models can also be used as targets. Here's a list of US cities named for presidents:
How it works
Marvin instructs the LLM to generate a list of JSON objects that satisfy the provided schema and instructions. Care is taken to introduce variation in the output, so that the samples are not all identical.
generate function is the primary tool for generating synthetic data. It accepts a
type argument, which can be any Python type, Pydantic model, or
Literal. It also has an argument
n, which specifies the number of samples to generate. Finally, it accepts an
instructions argument, which is a natural language description of the desired output. The LLM will use these instructions, in addition to the provided type, to guide its generation process. Instructions are especially important for types that are not self documenting, such as Python builtins like
generate supports almost all builtin Python types, plus Pydantic models, Python's
TypedDict. Pydantic models are especially useful for specifying specific features of the generated data, such as locations, dates, or more complex types. Builtin types are most useful in conjunction with instructions that provide more precise criteria for generation.
To specify the output type, pass it as the
target argument to
generate. The function will always return a list of
n items of the specified type. If no target is provided,
generate will return a list of strings.
OpenAI models currently have trouble parsing the API representation of tuples. Therefore we recommend using lists or Pydantic models (for more strict typing) instead. Tuple support will be added in a future release.
Data generation relies even more on instructions than other Marvin tools, as the potential for variation is much greater. Therefore, you should provide as much detail as possible in your instructions, in addition to any implicit documentation in your requested type.
Instructions are freeform natural language and can be as general or specific as you like. The LLM will do its best to comply with any instructions you give.
You can pass parameters to the underlying API via the
model_kwargs argument of
generate. These parameters are passed directly to the API, so you can use any supported parameter.
generate call would be independent. For some prompts, this would mean that each call produced very similar results to other calls. That would mean that generating, say, 10 items in a single call would produce a much more varied and high-quality result than generating 10 items in 5 calls of 2 items each.
To mediate this issue, Marvin maintains an in-memory cache of the last 100 results produced by each
generate prompt. These responses are shown to the LLM during generation to encourage variation. Note that the cache is not persisted across Python sessions. Cached results are also subject to a token cap to avoid flooding the LLM's context window. The token cap can be set with
MARVIN_AI_TEXT_GENERATE_CACHE_TOKEN_CAP and defaults to 600.
To disable this behavior, pass
Here is an example of how the cache improves generation. The first tab shows 10 cities generated in a single call; the second shows 10 cities generated in 5 calls of 2 cities each; and the third shows 10 cities generated in 5 calls but with the cache disabled.
The first and second tabs both show high-quality, varied results. The third tab is more disappointing, as it shows almost no variation.
Generate 10 cities in a single call, which produces a varied list:
Generate 10 cities in a five calls, using the cache. This also produces a varied list:
Generate 10 cities in five calls, without the cache. This produces a list with almost no variation, since each call is independent:
If you are using Marvin in an async environment, you can use