(no commit message)
This commit is contained in:
18
LICENSE
18
LICENSE
@@ -1,18 +0,0 @@
|
|||||||
MIT License
|
|
||||||
|
|
||||||
Copyright (c) 2026 probe_user
|
|
||||||
|
|
||||||
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
|
|
||||||
associated documentation files (the "Software"), to deal in the Software without restriction, including
|
|
||||||
without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
||||||
copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the
|
|
||||||
following conditions:
|
|
||||||
|
|
||||||
The above copyright notice and this permission notice shall be included in all copies or substantial
|
|
||||||
portions of the Software.
|
|
||||||
|
|
||||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
|
|
||||||
LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO
|
|
||||||
EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
|
||||||
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
|
|
||||||
USE OR OTHER DEALINGS IN THE SOFTWARE.
|
|
||||||
554
README.md
554
README.md
@@ -1,2 +1,554 @@
|
|||||||
# my-preferences
|
# Modaic Server - FastAPI Backend
|
||||||
|
|
||||||
|
**AI Agent Platform Backend Service**
|
||||||
|
|
||||||
|
This is the FastAPI backend server for the Modaic platform, providing a comprehensive REST API for AI agent management, user authentication, Git repository operations, and collaborative features.
|
||||||
|
|
||||||
|
   
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- **Python 3.13** or higher
|
||||||
|
- **UV** package manager
|
||||||
|
- **PostgreSQL** database
|
||||||
|
- **Stytch** account for authentication
|
||||||
|
- **AWS S3** bucket for file storage
|
||||||
|
- **Gitea** server for Git operations
|
||||||
|
|
||||||
|
### Installation
|
||||||
|
|
||||||
|
1. **Navigate to server directory**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd server
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Install dependencies**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv sync
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Set up environment variables**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
# Edit .env with your actual values
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Start the development server**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run uvicorn src.main:app --reload
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Access the API**
|
||||||
|
- **API Server**: http://localhost:8000
|
||||||
|
- **Interactive Docs**: http://localhost:8000/docs
|
||||||
|
- **ReDoc**: http://localhost:8000/redoc
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Application Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
src/
|
||||||
|
api/ # API endpoints organized by feature
|
||||||
|
v1/
|
||||||
|
auth/ # Authentication endpoints
|
||||||
|
agent/ # Agent management endpoints
|
||||||
|
user/ # User management endpoints
|
||||||
|
tokens/ # API token management
|
||||||
|
core/ # Core configuration
|
||||||
|
config.py # Application settings
|
||||||
|
db/ # Database configuration
|
||||||
|
index.py # Database connection
|
||||||
|
pg.py # PostgreSQL setup
|
||||||
|
lib/ # External service integrations
|
||||||
|
gitea.py # Gitea API client
|
||||||
|
s3.py # AWS S3 operations
|
||||||
|
stytch.py # Stytch authentication
|
||||||
|
logger.py # Logging configuration
|
||||||
|
objects/ # Data models and schemas
|
||||||
|
models/ # SQLAlchemy models
|
||||||
|
schemas/ # Pydantic schemas
|
||||||
|
service/ # Business logic services
|
||||||
|
email.py # Email service
|
||||||
|
utils/ # Utility functions
|
||||||
|
date.py # Date utilities
|
||||||
|
gitea.py # Gitea helpers
|
||||||
|
user.py # User utilities
|
||||||
|
main.py # FastAPI application entry point
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Components
|
||||||
|
|
||||||
|
**FastAPI Application** (`src/main.py`)
|
||||||
|
|
||||||
|
- CORS configuration for frontend integration
|
||||||
|
- API route registration
|
||||||
|
- Middleware setup
|
||||||
|
- Error handling
|
||||||
|
|
||||||
|
**Database Layer** (`src/db/`)
|
||||||
|
|
||||||
|
- PostgreSQL connection with SQLAlchemy
|
||||||
|
- Async database operations
|
||||||
|
- Connection pooling
|
||||||
|
|
||||||
|
**Authentication** (`src/lib/stytch.py`)
|
||||||
|
|
||||||
|
- Stytch integration for user management
|
||||||
|
- JWT token validation
|
||||||
|
- Session management
|
||||||
|
|
||||||
|
**Git Operations** (`src/lib/gitea.py`)
|
||||||
|
|
||||||
|
- Gitea API integration
|
||||||
|
- Repository CRUD operations
|
||||||
|
- File content management
|
||||||
|
- Branch and commit operations
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
### Authentication (`/api/v1/auth`)
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
| ------ | --------------- | ----------------------------- |
|
||||||
|
| POST | `/register` | User registration with Stytch |
|
||||||
|
| POST | `/login` | User authentication |
|
||||||
|
| GET | `/authenticate` | OAuth callback handling |
|
||||||
|
| GET | `/me` | Get current user information |
|
||||||
|
| POST | `/onboarding` | Complete user onboarding |
|
||||||
|
| DELETE | `/session` | User logout |
|
||||||
|
|
||||||
|
### User Management (`/api/v1/user`)
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
| ------ | ---------------------- | --------------------- |
|
||||||
|
| GET | `/{userId}` | Get user by ID |
|
||||||
|
| GET | `/username/{username}` | Get user by username |
|
||||||
|
| PUT | `/{userId}` | Update user profile |
|
||||||
|
| DELETE | `/{userId}` | Delete user account |
|
||||||
|
| GET | `/check/email/` | Check email existence |
|
||||||
|
|
||||||
|
### Agent Management (`/api/v1/agents`)
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
| ------ | --------------------------------------- | -------------------- |
|
||||||
|
| GET | `/user/{username}` | Get user's agents |
|
||||||
|
| POST | `/create` | Create new agent |
|
||||||
|
| GET | `/user/{username}/agent/{name}` | Get agent details |
|
||||||
|
| GET | `/search` | Search public agents |
|
||||||
|
| POST | `/star/owner/{username}/agent/{name}` | Star an agent |
|
||||||
|
| DELETE | `/unstar/owner/{username}/agent/{name}` | Unstar an agent |
|
||||||
|
|
||||||
|
### Repository Operations
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
| ------ | ----------------------------------------------- | ----------------- |
|
||||||
|
| GET | `/user/{username}/agent/{name}/contents/{path}` | Get file contents |
|
||||||
|
| GET | `/user/{username}/agent/{name}/raw/{path}` | Get raw file |
|
||||||
|
| GET | `/user/{username}/agent/{name}/branches` | List branches |
|
||||||
|
| GET | `/user/{username}/agent/{name}/commits` | Get commits |
|
||||||
|
| GET | `/user/{username}/agent/{name}/git/trees/{sha}` | Get git tree |
|
||||||
|
|
||||||
|
### Token Management (`/api/v1/tokens`)
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
| ------ | ---------------------------------- | ---------------- |
|
||||||
|
| POST | `/user/{user_id}` | Create API token |
|
||||||
|
| DELETE | `/user/{user_id}/token/{token_id}` | Delete token |
|
||||||
|
| GET | `/user/{user_id}` | Get user tokens |
|
||||||
|
|
||||||
|
## Database Models
|
||||||
|
|
||||||
|
### Core Models
|
||||||
|
|
||||||
|
**User Model**
|
||||||
|
|
||||||
|
```python
|
||||||
|
class User(Base):
|
||||||
|
userId: str # Primary key from Stytch
|
||||||
|
username: str # Unique username
|
||||||
|
email: str # User email
|
||||||
|
fullName: str # Display name
|
||||||
|
profilePictureUrl: str
|
||||||
|
bio: str
|
||||||
|
# Gitea integration fields
|
||||||
|
giteaUserId: int
|
||||||
|
giteaPassword: str # Encrypted
|
||||||
|
giteaToken: str # Encrypted
|
||||||
|
# Social links
|
||||||
|
githubUrl: str
|
||||||
|
linkedinUrl: str
|
||||||
|
xUrl: str
|
||||||
|
websiteUrl: str
|
||||||
|
```
|
||||||
|
|
||||||
|
**Agent Model**
|
||||||
|
|
||||||
|
```python
|
||||||
|
class Agent(Base):
|
||||||
|
agentId: str # Primary key
|
||||||
|
name: str # Agent name
|
||||||
|
description: str # Agent description
|
||||||
|
adminId: str # Owner user ID
|
||||||
|
configYaml: str # Agent configuration
|
||||||
|
readmeContent: str # README content
|
||||||
|
version: str # Agent version
|
||||||
|
visibility: str # public/private
|
||||||
|
created: datetime
|
||||||
|
updated: datetime
|
||||||
|
```
|
||||||
|
|
||||||
|
**Supporting Models**
|
||||||
|
|
||||||
|
- **Star**: User-Agent many-to-many relationship
|
||||||
|
- **Fork**: Fork tracking with original/forked agent IDs
|
||||||
|
- **Contributor**: Repository collaboration management
|
||||||
|
- **ImageKey**: S3 asset management
|
||||||
|
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
**Required Variables**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Authentication
|
||||||
|
STYTCH_PROJECT_ID="your-project-id"
|
||||||
|
STYTCH_SECRET="your-secret-key"
|
||||||
|
STYTCH_PROJECT_DOMAIN="your-domain"
|
||||||
|
|
||||||
|
# Database
|
||||||
|
POSTGRES_CONNECTION_URL="postgresql://user:pass@host:port/db"
|
||||||
|
POSTGRES_DATABASE="modaic_db"
|
||||||
|
|
||||||
|
# Gitea
|
||||||
|
GITEA_URL="http://localhost:4000"
|
||||||
|
GITEA_ADMIN_TOKEN="admin-token"
|
||||||
|
GITEA_ADMIN_USERNAME="admin"
|
||||||
|
GITEA_ADMIN_PASSWORD="password"
|
||||||
|
|
||||||
|
# AWS S3
|
||||||
|
S3_BUCKET_NAME="your-bucket"
|
||||||
|
CLOUDFRONT_DOMAIN="your-cloudfront-domain"
|
||||||
|
|
||||||
|
# Email
|
||||||
|
GMAIL_APP_PASSWORD="app-password"
|
||||||
|
|
||||||
|
# Security
|
||||||
|
ENCRYPTION_KEY="base64-encoded-key"
|
||||||
|
GUEST_TOKEN="guest-access-token"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Optional Variables**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Development
|
||||||
|
DEBUG="true"
|
||||||
|
LOG_LEVEL="info"
|
||||||
|
CACHE_DIR="./cache"
|
||||||
|
MAX_FILE_SIZE="10485760"
|
||||||
|
|
||||||
|
# Performance
|
||||||
|
CORS_ORIGINS="http://localhost:3000"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Configuration
|
||||||
|
|
||||||
|
The application uses SQLAlchemy with PostgreSQL:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Connection URL format
|
||||||
|
postgresql://username:password@host:port/database?sslmode=require
|
||||||
|
|
||||||
|
# Connection pooling (optional)
|
||||||
|
DB_POOL_SIZE=10
|
||||||
|
DB_MAX_OVERFLOW=20
|
||||||
|
DB_POOL_TIMEOUT=30
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security Features
|
||||||
|
|
||||||
|
### Authentication & Authorization
|
||||||
|
|
||||||
|
**Multi-layer Authentication**
|
||||||
|
|
||||||
|
- Bearer token authentication (API access)
|
||||||
|
- Session cookie authentication (browser)
|
||||||
|
- Stytch JWT validation
|
||||||
|
|
||||||
|
**Permission Levels**
|
||||||
|
|
||||||
|
- Public endpoints (no auth required)
|
||||||
|
- Authenticated user endpoints
|
||||||
|
- Owner-only operations
|
||||||
|
- Admin-level operations
|
||||||
|
|
||||||
|
### Data Protection
|
||||||
|
|
||||||
|
**Encryption**
|
||||||
|
|
||||||
|
- Gitea tokens encrypted with Fernet
|
||||||
|
- Sensitive user data encrypted at rest
|
||||||
|
- Secure password hashing
|
||||||
|
|
||||||
|
**Input Validation**
|
||||||
|
|
||||||
|
- Comprehensive Pydantic schemas
|
||||||
|
- SQL injection prevention
|
||||||
|
- XSS protection through escaping
|
||||||
|
|
||||||
|
**Security Headers**
|
||||||
|
|
||||||
|
- CORS configuration
|
||||||
|
- Rate limiting (configurable)
|
||||||
|
- Secure cookie settings
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Running Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install test dependencies
|
||||||
|
uv sync --group dev
|
||||||
|
|
||||||
|
# Run all tests
|
||||||
|
uv run pytest
|
||||||
|
|
||||||
|
# Run with coverage
|
||||||
|
uv run pytest --cov=src
|
||||||
|
|
||||||
|
# Run specific test file
|
||||||
|
uv run pytest tests/test_auth.py
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
### Code Quality
|
||||||
|
|
||||||
|
**Linting and Formatting**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Format code with Black
|
||||||
|
uv run black src/
|
||||||
|
|
||||||
|
# Lint with Ruff
|
||||||
|
uv run ruff check src/
|
||||||
|
|
||||||
|
# Type checking (if using mypy)
|
||||||
|
uv run mypy src/
|
||||||
|
```
|
||||||
|
|
||||||
|
**Pre-commit Hooks**
|
||||||
|
The project uses Ruff for linting and Black for formatting:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[tool.ruff]
|
||||||
|
line-length = 88
|
||||||
|
target-version = "py313"
|
||||||
|
select = ["E", "W", "F", "I", "B", "C4", "UP", "PD"]
|
||||||
|
|
||||||
|
[tool.ruff.format]
|
||||||
|
quote-style = "double"
|
||||||
|
indent-style = "space"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Operations
|
||||||
|
|
||||||
|
**Migrations** (if using Alembic)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate migration
|
||||||
|
alembic revision --autogenerate -m "description"
|
||||||
|
|
||||||
|
# Apply migrations
|
||||||
|
alembic upgrade head
|
||||||
|
|
||||||
|
# Downgrade
|
||||||
|
alembic downgrade -1
|
||||||
|
```
|
||||||
|
|
||||||
|
**Database Reset** (development only)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Drop and recreate tables
|
||||||
|
python -c "from src.db import recreate_tables; recreate_tables()"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
### Production Setup
|
||||||
|
|
||||||
|
**Environment Configuration**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set production environment
|
||||||
|
ENVIRONMENT="production"
|
||||||
|
|
||||||
|
# Use production database
|
||||||
|
POSTGRES_CONNECTION_URL="postgresql://prod_user:secure_pass@prod_host:5432/prod_db?sslmode=require"
|
||||||
|
|
||||||
|
# Configure secure domains
|
||||||
|
CORS_ORIGINS="https://yourdomain.com,https://app.yourdomain.com"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Docker Deployment**
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
FROM python:3.13-slim
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY pyproject.toml uv.lock ./
|
||||||
|
RUN pip install uv && uv sync --frozen
|
||||||
|
|
||||||
|
COPY src/ ./src/
|
||||||
|
CMD ["uv", "run", "uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Performance Optimization
|
||||||
|
|
||||||
|
**Database Optimization**
|
||||||
|
|
||||||
|
- Connection pooling configuration
|
||||||
|
- Query optimization with proper indexes
|
||||||
|
- Database query monitoring
|
||||||
|
|
||||||
|
**Caching Strategy**
|
||||||
|
|
||||||
|
- Response caching for read-heavy endpoints
|
||||||
|
- Redis integration (optional)
|
||||||
|
- CDN integration for static assets
|
||||||
|
|
||||||
|
## Monitoring & Logging
|
||||||
|
|
||||||
|
### Logging Configuration
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Structured logging
|
||||||
|
LOG_LEVEL="info" # debug, info, warning, error
|
||||||
|
|
||||||
|
# Log format includes:
|
||||||
|
# - Timestamp
|
||||||
|
# - Log level
|
||||||
|
# - Request ID
|
||||||
|
# - User ID (if authenticated)
|
||||||
|
# - Endpoint
|
||||||
|
# - Response time
|
||||||
|
```
|
||||||
|
|
||||||
|
### Health Checks
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Health check endpoint
|
||||||
|
GET /health
|
||||||
|
|
||||||
|
# Database connectivity check
|
||||||
|
GET /health/db
|
||||||
|
|
||||||
|
# External services check
|
||||||
|
GET /health/services
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
**Database Connection Errors**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check PostgreSQL status
|
||||||
|
pg_isready -h localhost -p 5432
|
||||||
|
|
||||||
|
# Test connection string
|
||||||
|
python -c "from src.db import engine; print(engine.connect())"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Stytch Authentication Issues**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Verify Stytch credentials
|
||||||
|
curl -X GET "https://test.stytch.com/v1/users" \
|
||||||
|
-H "Authorization: Bearer YOUR_SECRET_KEY"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Gitea Integration Problems**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Test Gitea API connectivity
|
||||||
|
curl -H "Authorization: token YOUR_GITEA_TOKEN" \
|
||||||
|
http://localhost:4000/api/v1/user
|
||||||
|
```
|
||||||
|
|
||||||
|
**File Upload Issues**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check S3 permissions
|
||||||
|
aws s3 ls s3://your-bucket-name
|
||||||
|
|
||||||
|
# Verify file size limits
|
||||||
|
MAX_FILE_SIZE=10485760 # 10MB
|
||||||
|
```
|
||||||
|
|
||||||
|
### Debug Mode
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start with debug logging
|
||||||
|
DEBUG=true uv run uvicorn src.main:app --reload --log-level debug
|
||||||
|
|
||||||
|
# Enable SQL query logging
|
||||||
|
echo "SQLALCHEMY_ECHO=true" >> .env
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Documentation
|
||||||
|
|
||||||
|
The FastAPI application automatically generates interactive API documentation:
|
||||||
|
|
||||||
|
- **Swagger UI**: http://localhost:8000/docs
|
||||||
|
- **ReDoc**: http://localhost:8000/redoc
|
||||||
|
- **OpenAPI JSON**: http://localhost:8000/openapi.json
|
||||||
|
|
||||||
|
### API Client Examples
|
||||||
|
|
||||||
|
**Python Client**
|
||||||
|
|
||||||
|
```python
|
||||||
|
import requests
|
||||||
|
|
||||||
|
# Authentication
|
||||||
|
response = requests.post("http://localhost:8000/api/v1/auth/login",
|
||||||
|
json={"email": "user@example.com", "password": "password"})
|
||||||
|
|
||||||
|
# Get user agents
|
||||||
|
response = requests.get("http://localhost:8000/api/v1/agents/user/username",
|
||||||
|
headers={"Authorization": f"Bearer {token}"})
|
||||||
|
```
|
||||||
|
|
||||||
|
**JavaScript Client**
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// Authentication
|
||||||
|
const response = await fetch('http://localhost:8000/api/v1/auth/login', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ email: 'user@example.com', password: 'password' }),
|
||||||
|
});
|
||||||
|
|
||||||
|
// Get agents
|
||||||
|
const agents = await fetch(
|
||||||
|
'http://localhost:8000/api/v1/agents/user/username',
|
||||||
|
{
|
||||||
|
headers: { Authorization: `Bearer ${token}` },
|
||||||
|
}
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**For more information, see the [main project README](../README.md)**
|
||||||
|
|||||||
52
config.json
Normal file
52
config.json
Normal file
@@ -0,0 +1,52 @@
|
|||||||
|
{
|
||||||
|
"model": null,
|
||||||
|
"signature": {
|
||||||
|
"description": "Evaluate and compare the quality of two responses (Response A and Response B) given a specific question.\nDetermine which response better addresses the question by focusing on factual correctness, completeness,\nand adherence to any specific requirements mentioned in the question prompt.\n\nBefore yielding your decision, think step by step and explain your reasoning in the reasoning field.\nBe sure to verbally express your uncertainty in your thought process.\n\nDetailed Instructions:\n\n1. **Understand the Question Context:**\n - Ensure you comprehend the full context and requirements specified by the question or problem statement.\n - Note any domain-specific terminologies or conditions.\n\n2. **Evaluate Each Response:**\n - Check for factual accuracy in the content, calculations, or recommendations provided.\n - Assess the response for completeness\u2014whether it completely addresses all aspects of the question.\n - Verify adherence to the specified question requirements.\n - Consider clarity and structure of the explanation or solution provided.\n\n3. **Decision Making:**\n - Determine which response (A or B) best meets the above criteria.\n - Select the response that is not only correct but also most aligns with the question's specific requirements.\n\n4. **Output Your Conclusion:**\n - Document your reasoning process in the reasoning field.\n - Output \"A>B\" if Response A is better, or \"B>A\" if Response B is better.",
|
||||||
|
"properties": {
|
||||||
|
"question": {
|
||||||
|
"__dspy_field_type": "input",
|
||||||
|
"desc": "The original question or prompt",
|
||||||
|
"prefix": "Question:",
|
||||||
|
"title": "Question",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"response_A": {
|
||||||
|
"__dspy_field_type": "input",
|
||||||
|
"desc": "First response to evaluate",
|
||||||
|
"prefix": "Response A:",
|
||||||
|
"title": "Response A",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"response_B": {
|
||||||
|
"__dspy_field_type": "input",
|
||||||
|
"desc": "Second response to evaluate",
|
||||||
|
"prefix": "Response B:",
|
||||||
|
"title": "Response B",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"reasoning": {
|
||||||
|
"__dspy_field_type": "output",
|
||||||
|
"desc": "Your step by step reasoning for why you chose the better response. With verbally expressed uncertainty.",
|
||||||
|
"prefix": "Reasoning:",
|
||||||
|
"title": "Reasoning",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"label": {
|
||||||
|
"__dspy_field_type": "output",
|
||||||
|
"desc": "Which response is better: 'A>B' or 'B>A'",
|
||||||
|
"prefix": "Label:",
|
||||||
|
"title": "Label",
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"question",
|
||||||
|
"response_A",
|
||||||
|
"response_B",
|
||||||
|
"reasoning",
|
||||||
|
"label"
|
||||||
|
],
|
||||||
|
"title": "PreferenceSig",
|
||||||
|
"type": "object"
|
||||||
|
}
|
||||||
|
}
|
||||||
1
probe.json
Normal file
1
probe.json
Normal file
@@ -0,0 +1 @@
|
|||||||
|
{"probe_version":"v1","embedding_dim":5120,"dropout":0.0,"layer_index":16,"num_layers":65,"probe_type":"linear"}
|
||||||
BIN
probe.safetensors
Normal file
BIN
probe.safetensors
Normal file
Binary file not shown.
48
program.json
Normal file
48
program.json
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
{
|
||||||
|
"traces": [],
|
||||||
|
"train": [],
|
||||||
|
"demos": [],
|
||||||
|
"signature": {
|
||||||
|
"instructions": "Evaluate and compare the quality of two responses (Response A and Response B) given a specific question.\nDetermine which response better addresses the question by focusing on factual correctness, completeness,\nand adherence to any specific requirements mentioned in the question prompt.\n\nBefore yielding your decision, think step by step and explain your reasoning in the reasoning field.\nBe sure to verbally express your uncertainty in your thought process.\n\nDetailed Instructions:\n\n1. **Understand the Question Context:**\n - Ensure you comprehend the full context and requirements specified by the question or problem statement.\n - Note any domain-specific terminologies or conditions.\n\n2. **Evaluate Each Response:**\n - Check for factual accuracy in the content, calculations, or recommendations provided.\n - Assess the response for completeness\u2014whether it completely addresses all aspects of the question.\n - Verify adherence to the specified question requirements.\n - Consider clarity and structure of the explanation or solution provided.\n\n3. **Decision Making:**\n - Determine which response (A or B) best meets the above criteria.\n - Select the response that is not only correct but also most aligns with the question's specific requirements.\n\n4. **Output Your Conclusion:**\n - Document your reasoning process in the reasoning field.\n - Output \"A>B\" if Response A is better, or \"B>A\" if Response B is better.",
|
||||||
|
"fields": [
|
||||||
|
{
|
||||||
|
"prefix": "Question:",
|
||||||
|
"description": "The original question or prompt"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"prefix": "Response A:",
|
||||||
|
"description": "First response to evaluate"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"prefix": "Response B:",
|
||||||
|
"description": "Second response to evaluate"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"prefix": "Reasoning:",
|
||||||
|
"description": "Your step by step reasoning for why you chose the better response. With verbally expressed uncertainty."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"prefix": "Label:",
|
||||||
|
"description": "Which response is better: 'A>B' or 'B>A'"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"lm": {
|
||||||
|
"model": "together_ai/Qwen/Qwen3-VL-32B-Instruct",
|
||||||
|
"model_type": "chat",
|
||||||
|
"cache": true,
|
||||||
|
"num_retries": 3,
|
||||||
|
"finetuning_model": null,
|
||||||
|
"launch_kwargs": {},
|
||||||
|
"train_kwargs": {},
|
||||||
|
"temperature": null,
|
||||||
|
"max_tokens": null
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"dependency_versions": {
|
||||||
|
"python": "3.13",
|
||||||
|
"dspy": "3.1.3",
|
||||||
|
"cloudpickle": "3.1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user