Response formats#

You can select from existing response formats or write a custom one. Any of the following can be used by name passed to the format key in the task.yml or info.yml for folder-based tasks.

Writing a Custom one#

We recommend building off of Pydantic BaseModel

Builtin#

class benchtools.response.Binary(*, answer: YesNoEnum)#

model_config = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class benchtools.response.BinaryJustification(*, answer: YesNoEnum, justification: str)#

model_config = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class benchtools.response.FloatAnswer(*, answer: float)#

model_config = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class benchtools.response.IntAnswer(*, answer: int)#

model_config = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class benchtools.response.IntJustification(*, answer: int, justification: str)#

model_config = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class benchtools.response.SentimentEnum(*values)#

class benchtools.response.StringAnswer(*, answer: str)#

simple string in an ‘answer’ key

model_config = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class benchtools.response.StringJustification(*, answer: str, justification: str)#

model_config = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class benchtools.response.YesNoEnum(*values)#