Official Technical Specifications

2.1 Complete Directory Structure

During evaluation, the complete directory structure inside the container is as follows. The participant's Agent must traverse all task subdirectories under /input, process them sequentially, and write results to the corresponding /output/task_<id>/ directory.

/input/                          # Read-only mount, no writes allowed
└── task_<id>/                   # One subdirectory per task, id is unique identifier
    ├── task.json                # Task metadata (task_id, difficulty, question)
    └── context/                 # Context data required for the task
        ├── csv/                 # Optional: Structured CSV files
        ├── db/                  # Optional: SQLite database files
        ├── json/                # Optional: Structured JSON files
        ├── doc/                 # Optional: Data documentation (Markdown, etc.)
        └── knowledge.md         # Optional: Background knowledge document

/output/                         # Read-write, participants write prediction results
└── task_<id>/                   # Corresponds one-to-one with input task directory
    └── prediction.csv           # [REQUIRED OUTPUT] Participant's prediction answer

2.2 task.json Format

The task metadata for each task is stored in task.json, with the following field descriptions:

task.json example:

{
  "task_id": "task_11",
  "difficulty": "medium",
  "question": "Which product category generated the highest total revenue in Q3? List each category and its total revenue, sorted by revenue descending."
}

2.3 context/ Data Sources Description

The data source types under the context/ directory vary by task. The participant's Agent must detect and read the subdirectories and files that actually exist:

Correspondence between difficulty and data sources:

2.4 Output Format: prediction.csv

For each task, the participant's Agent must write prediction results to /output/task_<id>/prediction.csv. This file is in standard CSV format (UTF-8 encoding) with the following requirements:

• The first row is the column names (header); column names do not participate in scoring and are only for readability
• From the second row onwards are data rows, each corresponding to one result record
• Column order does not affect scoring; scoring is based on unordered matching of column value vectors (see Section 6)
• Numeric data should retain sufficient precision; the evaluation system will normalize values to 2 decimal places (rounding) before comparison
• Null values should be written as empty strings; participants can write NULL, null, NaN, etc., and the evaluation system will normalize these to empty strings
• For name fields, splitting first name and last name into two columns or combining them into one full name column are both acceptable; the evaluation system considers both forms correct

prediction.csv example (corresponding question: list total sales by category):

category,total_revenue
Electronics,4200000.00
Clothing,1850000.00
Food,930000.00

3.1 Image Naming Convention

To ensure unique traceability of submitted images, participants must strictly follow the format below to name Docker images and compressed archives:

Example (Team ID is team0042, 3rd submission):

• Image name: team0042:v3
• Archive filename: team0042_v3.tar.gz

3.2 Image Content Requirements

3.3 Image Submission Process

Step 1: Upload image archive to Google Drive

• Upload <team_id>_v<N>.tar.gz to any location in your Google Drive.
• Right-click the file → "Share" → "Anyone with the link" → Set permission to "Viewer" → Copy link.

Step 2: Send submission email

Send the following information to the organizers' email (email address: [email protected]):

• Email subject: [KDDCup2026 Data Agents] Submission - <team_id> - v<N>
• Team ID (team_id)
• Submission version number (e.g., v3)
• Google Drive sharing link

Email example:

Subject: [KDDCup2026 Data Agents] Submission - team0000 - v1

Team ID: team0000
Version: v1
Sharing link: https://drive.google.com/file/d/1QTBRom51ejitPLe9Ke_HKi1PZkAyWKOg/view?usp=share_link

3.4 Evaluation System Execution Commands

After the image is uploaded, the evaluation system will execute the following steps in sequence:

Step 1: Decompress and load the image

docker load -i team0042_v3.tar.gz

Step 2: Start container for evaluation (conceptual representation, actual parameters generated by evaluation platform)

docker run --rm \
  --network=eval_net \           # Internal network (only access to evaluation system services)
  --cpus=16 \                    # CPU core limit
  --memory=64g \                 # Memory hard limit
  --memory-swap=64g \            # Disable swap space
  -v /eval/data/input:/input:ro \    # All task inputs (read-only)
  -v /eval/<submission_id>/output:/output:rw \  # All task outputs (read-write)
  -v /eval/<submission_id>/logs:/logs:rw \ # Solution runtime log files (read-write)
  -e MODEL_API_URL=<model_url> \      # Internal LLM model service URL
  -e MODEL_API_KEY=<api_key> \        # Internal LLM model API Key
  -e MODEL_NAME=<model_name> \        # Internal LLM model access name
  team0042:v3                         # Participant solution image name

3.5 Injected Environment Variables

3.6 Agent Core Logic Example

Core logic that main.py should implement (pseudocode):

import os, json
from pathlib import Path

input_root  = Path("/input")
output_root = Path("/output")

for task_dir in sorted(input_root.iterdir()):
    task_meta = json.loads((task_dir / "task.json").read_text())
    task_id   = task_meta["task_id"]
    question  = task_meta["question"]
    context   = task_dir / "context"

    # ... Run Agent, generate DataFrame result ...

    out_dir = output_root / task_id
    out_dir.mkdir(parents=True, exist_ok=True)
    result.to_csv(out_dir / "prediction.csv", index=False)

3.7 Runtime Log Specification

The evaluation system allocates an independent /logs directory (read-write) for each submission through Docker mounting. When a participant's solution exits abnormally, the organizers can extract the contents of the /logs directory for participant debugging reference.

Participants must ensure that /logs contains complete logs needed for troubleshooting, especially output when the program crashes. Since stdout/stderr are not automatically persisted after container exit, it is recommended to redirect both standard output and standard error to a log file in the startup command:

# Synchronously save stdout/stderr in ENTRYPOINT script or startup command
python main.py 2>&1 | tee /logs/runtime.log

4.1 Resource Quota

4.2 Timeout & OOM Handling

• Timeout (12 hours): The evaluation system sends SIGTERM and waits 30 seconds before sending SIGKILL if not exited. Already written prediction.csv files will still participate in scoring; unwritten tasks receive 0 points.
• OOM Kill: Container is forcibly terminated; already written prediction.csv files participate in scoring; unwritten tasks receive 0 points.
• Non-zero exit code: Does not affect scoring; the evaluation system only cares whether files exist in the /output directory.

5.1 Development Stage vs Evaluation Stage

This competition adopts a "develop freely, evaluate uniformly" strategy for model usage:

5.2 Runtime Environment Variable Injection

When starting participant containers, the evaluation system injects runtime environment variables in the form of docker run -e KEY=VALUE. Participant programs must read these configurations from environment variables at runtime and must not hardcode API URLs, API Keys, and other information in code or images.

5.3 Recommended Code Patterns

Method 1: Using OpenAI SDK (Recommended)

import os
from openai import OpenAI

# Read from environment variables—set your own values during local development,
# injected by system during evaluation
client = OpenAI(
    base_url=os.environ["MODEL_API_URL"],
    api_key=os.environ.get("MODEL_API_KEY", "EMPTY"),
)
model_name = os.environ.get("MODEL_NAME", "qwen3.5-35b-a3b")

response = client.chat.completions.create(
    model=model_name,
    messages=[
        {"role": "system", "content": "You are a data analysis agent."},
        {"role": "user",   "content": question},
    ],
    ...,
)
answer = response.choices[0].message.content

Method 2: Using requests to directly call HTTP interface

import os, requests

resp = requests.post(
    os.environ["MODEL_API_URL"] + "/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ.get('MODEL_API_KEY', 'EMPTY')}"},
    json={
        "model": os.environ.get("MODEL_NAME", "qwen3.5-35b-a3b"),
        "messages": [{"role": "user", "content": question}],
        "temperature": 0.0,
    },
    ...,
)
answer = resp.json()["choices"][0]["message"]["content"]

5.4 Network Access Policy

6.1 Evaluation Process Description

The evaluation system uses a unified automated evaluation program to perform offline batch scoring of participant submitted prediction results. Participants need to submit result files in the specified format (such as prediction.csv), and the system will compare them against corresponding standard answers for evaluation and generate final scores.

All submissions run in the same evaluation environment, and the evaluation process is deterministic execution to ensure consistency and fairness of evaluation results.

6.2 Column Matching Method

Evaluation uses a column-level content consistency matching method (column-level matching), with the core process as follows:

1. Read prediction.csv and gold.csv, parse columns and data content
2. For each column, sort all cell values in that column to construct a "column signature"
3. Count the number of occurrences of column signatures in prediction and gold respectively
4. Match based on column signatures, calculate prediction's coverage of gold

• Ignore column names, match only based on column content
• Ignore row order (implemented through sorting)
• Support duplicate columns (same column signature needs to match the same number of times)

6.3 Scoring Metric Calculation

Evaluation introduces a light penalty for redundant predictions based on coverage (Recall) to more comprehensively measure prediction result quality.

Definitions:

• Matched Columns: Number of columns in prediction that successfully match gold (matched by column signature)
• Gold Columns: Total number of columns in gold
• Predicted Columns: Total number of columns in prediction
• Extra Columns: Number of columns in prediction that do not match gold

First calculate:

Final score is:

Where:

• Penalty term λ weight is a preset constant, used to balance the relationship between coverage and redundant predictions
• When prediction results contain many unmatched columns, the score will decrease accordingly
• The lower bound of the final score is 0

6.4 Design Rationale

This scoring method moderately constrains the "redundancy level" of prediction results while maintaining a Recall orientation, with the following design goals:

• ✅ Encourage complete coverage: Prioritize evaluating whether all target columns are found
• ✅ Control redundant output: Penalize predictions containing many irrelevant columns
• ✅ Maintain robustness: Do not rely on column names or order, judge only based on data content

Evaluation not only focuses on "whether all are found" (Recall), but also encourages "as concise and accurate as possible" prediction results.

6.5 Value & Time Normalization Rules

Before constructing column signatures, the evaluation system normalizes cell content to reduce the impact of format differences (such as standardization of numeric and time types).

6.6 Scoring Examples

6.7 Total Score & Leaderboard

• Each task's score is calculated independently according to the above rules
• Total score is the average of all task scores
• Leaderboard is sorted by total score in descending order
• If total scores are the same, sort by last valid submission time in ascending order (earlier submission ranks higher)