Repository navigation

Multi-modal Optimizer + Context for Optimization - #50

Open

allenanie wants to merge 78 commits into

experimentalfrom

features/multimodal_opt

Open

Multi-modal Optimizer + Context for Optimization#50
allenanie wants to merge 78 commits into
experimentalfrom
features/multimodal_opt

Commits on Oct 3, 2025
- initial changes
  allenanie
  committed
  8961b02
- make context optional
  allenanie
  committed
  949d8ff
- finish adding image support to optoprime_v2
  allenanie
  committed
  e36dd7c
- Finish updating OPRO to accept additional context
  allenanie
  committed
  7dcb880
Commits on Oct 5, 2025
Commits on Oct 6, 2025
Commits on Nov 10, 2025
- add three types of image loading: from numpy array, from url, from local file.
  allenanie
  committed
  1791b15
- Merge branch 'experimental' into features/multimodal_opt
  allenanie
  authored
  502dd7a
- Update opto/optimizers/optoprime_v2.py
  
  allenanie
  and
  Copilot
  authored
  51dfc4b
- Initial plan
  Copilot
  committed
  978fdf6
- Merge pull request #54 from AgentOpt/copilot/sub-pr-50
  allenanie
  authored
  1fda4d7
- Update opto/optimizers/optoprime_v2.py
  
  allenanie
  and
  Copilot
  authored
  34f9747
- Update opto/optimizers/optoprime_v2.py
  
  allenanie
  and
  Copilot
  authored
  0118979
- Update opto/optimizers/opro_v2.py
  
  allenanie
  and
  Copilot
  authored
  a32136e
- Update opto/features/flows/types.py
  
  allenanie
  and
  Copilot
  authored
  52d5eb0
- Update opto/optimizers/optoprime_v2.py
  
  allenanie
  and
  Copilot
  authored
  d502eeb
- Update opto/features/flows/compose.py
  
  allenanie
  and
  Copilot
  authored
  e511a85
- Update opto/optimizers/optoprime_v2.py
  
  allenanie
  and
  Copilot
  authored
  2633ac7
- partial commit
  allenanie
  committed
  bbe1b40
- Merge branch 'features/multimodal_opt' of https://github.com/AgentOpt/OpenTrace into features/multimodal_opt
  allenanie
  committed
  d2fe092
Commits on Nov 21, 2025
- Merge branch 'experimental' of https://github.com/AgentOpt/OpenTrace into features/multimodal_opt
  allenanie
  committed
  ee6537c
Commits on Nov 22, 2025
- add a `is_image` check to Node
  allenanie
  committed
  a4057ef
- add parameter check to ensure 1 image can be parameter
  allenanie
  committed
  f6ce6b7
- fix the import error
  allenanie
  committed
  aa96807
Commits on Nov 26, 2025
- refactored/moved out image data check
  allenanie
  committed
  172d00f
Commits on Dec 1, 2025
- fix a few optimizer test issues, update setup dependencies to include pillow. Add history manager.
  allenanie
  committed
  40dfdab
- add protected rounds
  allenanie
  committed
  85edcdc
- add content block
  allenanie
  committed
  82a590d
Commits on Dec 2, 2025
- intermediate commit
  allenanie
  committed
  f3668fc
- update actual call to ensure API correctness
  allenanie
  committed
  6bace1c
- fix opro_v3 issues
  allenanie
  committed
  79b9d58
- update python dependency
  allenanie
  committed
  3d3f8a1
Commits on Dec 3, 2025
- update the tests (it was broken before)
  allenanie
  committed
  240f319
Commits on Dec 16, 2025
- add new workflow tests conditions to avoid long build time on experimental
  allenanie
  committed
  a077717
- partial refactoring. Reworked `add_context` to support interleaved image/text insertion.
  allenanie
  committed
  e01ddc8
Commits on Dec 17, 2025
- continued refactoring...
  allenanie
  committed
  a8a0c3a
- Adding prompt template to accommondate multi-modal fill-in
  allenanie
  committed
  7ff308d
- final update to both backbone and adding context.
  allenanie
  committed
  5ee4860
- updating to take feedback as image.
  allenanie
  committed
  f451250
- image as feedback is now roughly correct
  allenanie
  committed
  1122d8b
- remove not needed files
  allenanie
  committed
  781b7ce
- add back minibatch.ipynb
  allenanie
  committed
  095e7f7
Commits on Dec 22, 2025
- refactored LLMFactory -- removing "cheap", "premium" descriptions because these meanings are shifting (the "premium" model of 2025 will be the "cheap" model of 2027, which causes confusion and unre…
  allenanie
  committed
  7221e61
Commits on Dec 27, 2025
Commits on Jan 9, 2026
- fixed a few bugs, multi-turn multi-modal optimizer is runnable end-to-end now.
  allenanie
  committed
  d9231c1
- update tests to use the response API
  allenanie
  committed
  2ae6d10
- making the JSON extraction more robust. Adding a notebook for the demo.
  allenanie
  committed
  9ea88c9
- update oprov3
  allenanie
  committed
  a72b446
- move Content to backbone for better import to other optimziers
  allenanie
  committed
  125d9a2
Commits on Jan 13, 2026
- fix an Azure related bug
  allenanie
  committed
  4a2682a
- remove bug from __init__
  allenanie
  committed
  59f0f32
Commits on Jan 14, 2026
- add api_version explicitly to litellm
  allenanie
  committed
  301ee12
Commits on Jan 20, 2026
- Rename `ConversationHistory` to `Chat` because it is just a chat session management tool. Expanded some functionality on `Chat`.
  allenanie
  committed
  d4b536e
Commits on Jan 21, 2026
- Add jupyter visualization for multi-modal content rendering
  allenanie
  committed
  130b4fc
- fixed a few bugs
  allenanie
  committed
  be419cc
Commits on Jan 24, 2026
- update the import on jupyter notebook
  allenanie
  committed
  839a7b9
- have to keep `mm_beta=True` in llm.py so that Response API is the default because the `backbone.py` and other code already migrated. Can set `mm_beta=False` to go back to completion API.
  allenanie
  committed
  080e301
Commits on Feb 1, 2026
- updating the parsing logic for Amazon Bedrock response API setup
  allenanie
  committed
  adc405a
Commits on Feb 2, 2026
- adding aws bedrock JSON format exemption for all optimizers
  allenanie
  committed
  0e419b8
Commits on Feb 3, 2026
- add None exception to bedrock model name check
  allenanie
  committed
  46528bb
- fix the test case
  allenanie
  committed
  0cade07
- update the dummy LLM to make the test pass
  allenanie
  committed
  3cf760a
Commits on Feb 8, 2026
- Amelia added a fix to the newline continuation error after extracting code from JSON.
  allenanie
  committed
  c8e4d4f
Commits on Mar 24, 2026
- update names
  allenanie
  committed
  3eda73f
Commits on Apr 16, 2026
- refactor files
  xuanfeiren
  committed
  0e28b49
Commits on Apr 24, 2026
- updated the refactoring
  xuanfeiren
  committed
  8ea59da
- Added ParetobasedPS
  xuanfeiren
  committed
  5d2e98d
- update
  xuanfeiren
  committed
  9e02879
Commits on Apr 25, 2026
- update due to errors reported by Trace-Bench
  allenanie
  committed
  1714b6d
Commits on May 21, 2026
- Merge branch 'trainer_refactoring' of https://github.com/AgentOpt/OpenTrace into features/multimodal_opt
  allenanie
  committed
  78164dc

Commits

Commits on Oct 3, 2025

Commits on Oct 5, 2025

Commits on Oct 6, 2025

Commits on Nov 10, 2025

Commits on Nov 21, 2025

Commits on Nov 22, 2025

Commits on Nov 26, 2025

Commits on Dec 1, 2025

Commits on Dec 2, 2025

Commits on Dec 3, 2025

Commits on Dec 16, 2025

Commits on Dec 17, 2025

Commits on Dec 22, 2025

Commits on Dec 27, 2025

Commits on Jan 9, 2026

Commits on Jan 13, 2026

Commits on Jan 14, 2026

Commits on Jan 20, 2026

Commits on Jan 21, 2026

Commits on Jan 24, 2026

Commits on Feb 1, 2026

Commits on Feb 2, 2026

Commits on Feb 3, 2026

Commits on Feb 8, 2026

Commits on Mar 24, 2026

Commits on Apr 16, 2026

Commits on Apr 24, 2026

Commits on Apr 25, 2026

Commits on May 21, 2026