Improve response quality with Gemini. Improve evaluation harness #1150
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
debanjum
force-pushed
the
reduce-repetition-by-gemini-and-add-more-knobs-to-evals-workflow
branch
from
6be6107 to
9156a92
Compare
…iance. Gemini 2.0 flash model is cheaper and better than Gemini 1.5 pro
- Control auto read webpage via eval workflow. Prefix env var with KHOJ_
Default to false as it is the default that is going to be used in prod
going forward.
- Set openai api key via input param in manual eval workflow runs
- Simplify evaluating other chat models available over openai
compatible api via eval workflow.
- Mask input api key as secret in workflow.
- Discard unnecessary null setting of env vars.
- Control randomization of samples in eval workflow.
If randomization is turned off, it'll take the first SAMPLE_SIZE
items from the eval dataset instead of a random collection of
SAMPLE_SIZE items.
This is the default temperature for non-thinking gemini models on ai studio. See if using this alleviates the problem.
The queries field name in the first example isn't wrapped in double quotes, rest are.
Make research planner consistently select tool before query. As the model should tune it's query for the selected tool. It got space to think about tool to use in the scratchpad already.
We'd moved research planner to only use tools in enum of schema. This enum tool enforcement prevented model from terminating research by setting tool field to empty. Fix the issue by adding text tool to research tools enum and tell model to use that to terminate research and start response instead.
…odels Previously Gemini 2 flash and flash lite were using context window of 10K by default as no defaults were added for it. Increase default context for small commercial models to 120K from 60K as cheaper and faster than their pro models equivalents at 60K context.
debanjum
force-pushed
the
reduce-repetition-by-gemini-and-add-more-knobs-to-evals-workflow
branch
from
9156a92 to
7f18bc0
Compare
debanjum
changed the title
debanjum
deleted the
reduce-repetition-by-gemini-and-add-more-knobs-to-evals-workflow
branch
No Sign up for free
to join this conversation on GitHub.
Already have an account?
No Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Improve Gemini usage
2x context window for small, commercial models to 120K
Improve evaluation harness