Questions about task data descriptions and input-to-rubric mappings (seven tasks) #14

Hi maintainers,

Thank you for releasing ResearchClawBench and its task materials. We are studying the public AutoSciRub trajectories to better understand where scientific agents struggle, and encountered a few questions about how to interpret the supplied inputs and evaluation criteria.

We read the clarification in #10 that agents may obtain additional public data. That distinction is helpful: some of the cases below concern file descriptions or dimensions, while others may need a pointer to an intended mapping or external dataset. We would appreciate your guidance on the latter.

The checks below use commit 94dcebb528323a4637324f10f263e89ac31baef9. We have included the file links and a short reproduction script to make the observations easier to check.

File descriptions and dimensions

1. Chemistry_001: structure identity and atom coverage

• task_info.json describes FKBP12, 107 residues, CA-only coordinates, and an FK506 ligand.
• The supplied PDB contains 2,591 ATOM records and 161 CA records, all in chain A. Its CA residue identities and coordinates match chain A of the first NMR model in RCSB 2L3R, UHRF1 tandem Tudor domains bound to a histone H3 peptide.
• All 90 non-hydrogen atoms in the supplied ligand SDF match chain B of that RCSB model by element and coordinates rounded to 0.001 Å. That chain is an H3-derived peptide including trimethyllysine, rather than FK506. The SDF has 194 atoms including hydrogens.

We have proposed a small correction to these two file descriptions in #15. For the broader multi-complex training/docking evaluation, is there a recommended dataset or acquisition procedure that agents are expected to use?

2. Life_002: target structure and reference chain matching

Checklist item 0 requires nine simultaneous one-to-one chain matches between 7XG4 and 6N40.

• 7xg4.pdb: nine protein chains A B C D E F G H L (2,876 CA records).
• 6n40.pdb: one protein chain A (726 CA records). Its biological-assembly transform is the identity.
• RCSB 6N40 independently identifies a monomeric MmpL3 membrane protein, rather than a matching nine-subunit CRISPR complex.

Could you help confirm whether 6N40 is the intended target for this example? If a different multimeric target was intended, a corrected identifier would help us check the reference alignment. If this pair is intended as a negative or partial match, it would be useful to clarify how that relates to the nine-chain criterion.

3. Earth_003: sample grid resolution and forecast coverage

task_info.json says the input is (2,70,721,1440) at 0.25°, and describes 006.nc as forecasts at six-hour intervals up to 15 days.

The actual NetCDF headers show:

These headers can be reproduced with ncdump -h tasks/Earth_003/data/<filename>. A single six-hour forecast does not contain the 60 lead times described in the manifest. The package also contains no verifying observations or reference forecasts for the checklist's 2018 comparisons.

Are these files intended as lightweight 1° examples, with agents expected to obtain the full evaluation data separately? If so, a note in the manifest and a link to the intended data/model acquisition procedure would be very helpful. We would be happy to help with a small description update once the intended scope is clear.

4. Physics_001: coordinate–response array pairing

The core dataset is described as directly sufficient to reproduce all three experiments. Its labelled arrays have these lengths:

We could not find separate coordinate arrays or observation IDs in this file explaining those differences. Before fitting the exponent, we would like to understand the intended pairing rather than assume prefix truncation or a new grid. The arrays labelled BCS and power-law n=2 are also numerically identical in this file.

Is there a source table, generator, or coordinate convention for these curves that we may have missed? That would help us establish the correct pairings and interpret the two model labels.

Questions about data mappings and additional inputs

5. Material_001: graph/target serialization

The materials text file contains four unnamed lists in the property-prediction section, with lengths 100, 117, 20 and 97. The first contains only 5; the third contains integers 0–4 consistent with an edge template. We could not identify a schema in the supplied text specifying graph boundaries, per-node features, graph-to-target mapping, material IDs or target units. In particular, it is unclear what sample or node grouping the 117 scalar values represent.

The optimization section supplies two bounds and four singleton lists, but no labelled response table or objective function. We are therefore unsure which reaction response the optimization example is intended to use.

Could you point us to the intended graph/target schema and optimization objective, or the original example from which these lists were exported? A short mapping description may be enough to resolve this.

6. Life_001: patient mapping for the efficacy tables

The checklist asks for seven-patient efficacy distributions at 10,000 cells, and recall against experimentally validated neoantigens and 11 ranking baselines.

• The ten vaccine-elements.scores.100-cells.10x.rep-*.csv files each contain 100 cells × 12 elements, without patient identifiers.
• final-response-likelihoods.csv contains 1,000 rows grouped as 100-cells.10x, 0 through 100-cells.10x, 9, again without patient IDs.
• optimization_runtime_data.csv contains seven SampleIDs and 35 runtime rows. Its columns are SampleID,PopulationSize,RunTime; it supplies no join from those patient IDs to the response matrices.
• The supplied tables do not include experimental immunogenicity labels or the 11 baseline rankings needed for the recall comparison.

Is there a mapping from the efficacy replicates to the patient IDs, or an upstream dataset that agents are expected to retrieve for the seven-patient and recall analyses? We would appreciate a pointer so that our trajectory analysis uses the intended evaluation inputs.

7. Neuroscience_000: Together_1 sample and condition-specific criteria

The current task description asks for a supervised SimBA-style workflow on the Together_1 sample project. The supplied feature and target tables each have 1,738 rows; the auxiliary reference table has 300 rows. None contains lab, sex or RI/CSDS condition assignments.

Checklist items 0–3 require six-condition PR curves and Lab1/Lab2, male/female, RI/CSDS SHAP comparisons. Those comparisons cannot be constructed from the supplied sample without additional labelled cohorts and an assignment mapping.

Is the intended scope the current Together_1 sample-classification workflow, or should agents obtain the multi-cohort data used for those comparisons? Clarifying that scope and any cohort mapping would help us distinguish missing experimental coverage from missing input data. We are also unsure how the 300-row auxiliary reference relates to the training sequence.

Reproduction

from pathlib import Path
from collections import Counter
import ast, csv, re

r = Path('tasks')
for task in ('Chemistry_001', 'Life_002'):
    for p in sorted((r / task / 'data').rglob('*.pdb')):
        atoms = [s for s in p.read_text().splitlines() if s.startswith('ATOM  ')]
        ca = [s for s in atoms if s[12:16].strip() == 'CA']
        print(task, p.name, 'ATOM', len(atoms), 'CA', len(ca),
              'chains', dict(Counter(s[21] for s in ca)))

p = r / 'Physics_001/data/MATBG Superfluid Stiffness Core Dataset.txt'
for m in re.finditer(r'^\*\*(.+?):\*\*\s*\n(\[[^\]]*\])', p.read_text(), re.M):
    print('Physics_001', m[1], len(m[2][1:-1].split()))

p = r / 'Material_001/data/M-AI-Synth__Materials_AI_Dataset_.txt'
print('Material_001 list lengths',
      [len(ast.literal_eval(s)) for s in p.read_text().splitlines() if s.startswith('[')])

def read(p):
    with p.open(newline='') as f:
        return list(csv.DictReader(f))

d = r / 'Life_001/data'
a = read(d / 'vaccine-elements.scores.100-cells.10x.rep-0.csv')
print('Life_001 score columns', list(a[0]))
print('Life_001 cells/elements', len({v['cell_id'] for v in a}),
      len({v['vaccine_element'] for v in a}))
for name in ('final-response-likelihoods.csv', 'optimization_runtime_data.csv'):
    a = read(d / name)
    print('Life_001', name, len(a), list(a[0]))
for p in sorted((r / 'Neuroscience_000/data').glob('*.csv')):
    a = read(p)
    print('Neuroscience_000', p.name, len(a), list(a[0]))

For the mapping and acquisition questions, links to existing documentation or upstream data would already be very helpful. We can help with focused follow-up fixes once the intended inputs are confirmed. Thank you for taking a look.