Total tasks
142,283
Mean difficulty
5.89
Median difficulty
—
Avg patch lines
197.4
Avg patch files
4.84
Difficulty distribution All datasets
| Label breakdown | easy | medium | hard |
|---|---|---|---|
25808 / 68922 / 47551 |
25,808 | 68,922 | 47,551 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 142,281 | — | — | — | 5.89 | — | — |
Score bins
<=314,521 (10.2%)
3.1-525,371 (17.8%)
5.1-754,838 (38.5%)
7.1-839,681 (27.9%)
>87,870 (5.5%)
Languages
python75,943 (53.4%)
go13,659 (9.6%)
javascript11,024 (7.7%)
typescript10,494 (7.4%)
rust8,040 (5.7%)
c6,648 (4.7%)
java5,736 (4.0%)
cpp4,106 (2.9%)
php1,462 (1.0%)
kotlin881 (0.6%)
julia795 (0.6%)
scala460 (0.3%)
elixir411 (0.3%)
swift359 (0.3%)
dart282 (0.2%)
Area / tier
backend66,730 (46.9%)
library49,727 (34.9%)
frontend13,613 (9.6%)
cli12,166 (8.6%)
fullstack41 (0.0%)
framework4 (0.0%)
Top topics
react874 (4.3%)
sympy732 (3.6%)
pandas722 (3.5%)
kubernetes714 (3.5%)
django671 (3.3%)
click602 (3.0%)
xarray490 (2.4%)
sqlglot454 (2.2%)
eslint442 (2.2%)
pydantic434 (2.1%)
swc407 (2.0%)
matplotlib379 (1.9%)
fonttools375 (1.8%)
cobra349 (1.7%)
argparse321 (1.6%)
numpy307 (1.5%)
dask272 (1.3%)
geopandas272 (1.3%)
pennylane243 (1.2%)
qiskit235 (1.2%)
moto232 (1.1%)
clap231 (1.1%)
dvc215 (1.1%)
sphinx213 (1.0%)
globus-sdk213 (1.0%)
Bug classes
incomplete-validation10,626 (20.1%)
missing-feature6,838 (12.9%)
missing-fallback6,411 (12.1%)
type-handling-inconsistency6,202 (11.7%)
missing-metadata-propagation6,138 (11.6%)
wrong-default5,126 (9.7%)
missing-validation1,734 (3.3%)
missing-implementation1,257 (2.4%)
missing-state-propagation1,215 (2.3%)
incomplete-parsing799 (1.5%)
missing-method794 (1.5%)
missing-error-handling791 (1.5%)
race-condition764 (1.4%)
missing-functionality628 (1.2%)
missing-default584 (1.1%)
feature-addition407 (0.8%)
missing-api339 (0.6%)
index-out-of-bounds336 (0.6%)
missing-method-implementation268 (0.5%)
resource-leak209 (0.4%)
Methodology unified difficulty & tagging
Difficulty Scoring (1–10 scale)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Composite weighted score over 5 dimensions:
• Patch scope (30%): lines changed, files affected, hunks count
• Logic complexity (25%): control-flow depth, branching, algorithmic sophistication
• Context breadth (20%): cross-module dependencies, API surface understanding
• Test complexity (15%): fixture setup, mock requirements, edge-case coverage
• Instruction complexity (10%): problem-statement clarity, implicit requirements
Each dimension is log-scaled, then the weighted sum is mapped to 1–10 and binned into easy (≤4.0), medium (4.1–7.0), hard (>7.0).
Semantic Tagging
Each task is labelled with a 4-tuple
[language, area, topic, bug_class]:• language: primary programming language (python, javascript, go, …)
• area: architectural tier ∈ {backend, frontend, fullstack, cli, library, framework}
• topic: functional domain / library (auth, database, api, numpy, …)
• bug_class: root-cause category (logic-error, type-mismatch, race-condition, …)
Total tasks
45,881
Mean difficulty
6.30
Median difficulty
6.7
Avg patch lines
313.3
Avg patch files
6.41
Difficulty distribution LegoFlow-Instances
| Label breakdown | easy | medium | hard |
|---|---|---|---|
4617 / 22809 / 18455 |
4,617 | 22,809 | 18,455 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 45,881 | — | — | 6.7 | 6.30 | — | — |
Score bins
<=31,576 (3.4%)
3.1-56,648 (14.5%)
5.1-719,202 (41.9%)
7.1-818,455 (40.2%)
>80 (0.0%)
Languages
go7,487 (16.3%)
javascript6,592 (14.4%)
c6,466 (14.1%)
typescript6,426 (14.0%)
python5,535 (12.1%)
rust5,059 (11.0%)
java3,915 (8.5%)
cpp3,730 (8.1%)
ruby194 (0.4%)
vue102 (0.2%)
lua63 (0.1%)
php56 (0.1%)
solidity42 (0.1%)
yaml20 (0.0%)
cmake18 (0.0%)
Area / tier
backend24,598 (53.6%)
library10,855 (23.7%)
frontend6,639 (14.5%)
cli3,760 (8.2%)
fullstack27 (0.1%)
framework2 (0.0%)
Top topics
react606 (16.1%)
kubernetes284 (7.6%)
http197 (5.2%)
cobra196 (5.2%)
grpc161 (4.3%)
aem160 (4.3%)
aws-sdk145 (3.9%)
eslint145 (3.9%)
clap122 (3.2%)
mongoose122 (3.2%)
github-actions120 (3.2%)
git119 (3.2%)
electron117 (3.1%)
cmake117 (3.1%)
prometheus112 (3.0%)
pgbackrest110 (2.9%)
docker107 (2.8%)
redis106 (2.8%)
picoquic106 (2.8%)
igraph105 (2.8%)
openssl104 (2.8%)
fastify102 (2.7%)
svelte101 (2.7%)
express100 (2.7%)
ava96 (2.6%)
Bug classes
incomplete-validation3,075 (19.0%)
missing-feature2,405 (14.8%)
missing-fallback2,130 (13.1%)
missing-metadata-propagation1,723 (10.6%)
wrong-default1,627 (10.0%)
type-handling-inconsistency1,173 (7.2%)
missing-validation601 (3.7%)
missing-implementation507 (3.1%)
missing-state-propagation495 (3.1%)
race-condition407 (2.5%)
missing-api339 (2.1%)
missing-functionality273 (1.7%)
missing-error-handling241 (1.5%)
missing-method225 (1.4%)
resource-leak209 (1.3%)
incomplete-parsing207 (1.3%)
missing-config-option174 (1.1%)
missing-default148 (0.9%)
missing-function132 (0.8%)
feature-addition126 (0.8%)
Total tasks
21,336
Mean difficulty
5.64
Median difficulty
6.0
Avg patch lines
142.2
Avg patch files
3.87
Difficulty distribution SWE-rebench
| Label breakdown | easy | medium | hard |
|---|---|---|---|
5349 / 9133 / 6854 |
5,349 | 9,133 | 6,854 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 21,336 | — | — | 6.0 | 5.64 | — | — |
Score bins
<=33,346 (15.7%)
3.1-54,316 (20.2%)
5.1-76,820 (32.0%)
7.1-84,191 (19.6%)
>82,663 (12.5%)
Languages
python21,216 (99.4%)
fortran43 (0.2%)
cpp41 (0.2%)
javascript7 (0.0%)
sql5 (0.0%)
vyper4 (0.0%)
typescript3 (0.0%)
rust3 (0.0%)
java2 (0.0%)
bash2 (0.0%)
c2 (0.0%)
solidity2 (0.0%)
cython1 (0.0%)
terraform1 (0.0%)
cmake1 (0.0%)
Area / tier
backend9,601 (45.0%)
library8,992 (42.1%)
cli1,784 (8.4%)
frontend958 (4.5%)
fullstack1 (0.0%)
Top topics
sympy732 (15.0%)
sqlglot454 (9.3%)
xarray340 (7.0%)
matplotlib272 (5.6%)
dvc215 (4.4%)
sphinx213 (4.4%)
dask198 (4.1%)
pybamm191 (3.9%)
argparse172 (3.5%)
pandas172 (3.5%)
pillow170 (3.5%)
astropy162 (3.3%)
conan156 (3.2%)
numpy153 (3.1%)
sqlfluff139 (2.9%)
pydicom137 (2.8%)
click135 (2.8%)
geopandas124 (2.5%)
networkx122 (2.5%)
nilearn109 (2.2%)
pennylane108 (2.2%)
pytorch-ignite108 (2.2%)
tox104 (2.1%)
narwhals95 (2.0%)
textual89 (1.8%)
Bug classes
incomplete-validation1,752 (20.2%)
type-handling-inconsistency1,465 (16.9%)
missing-metadata-propagation1,042 (12.0%)
missing-fallback1,036 (11.9%)
wrong-default930 (10.7%)
missing-feature838 (9.7%)
missing-validation247 (2.8%)
incomplete-parsing171 (2.0%)
missing-implementation163 (1.9%)
missing-state-propagation151 (1.7%)
missing-method125 (1.4%)
missing-default116 (1.3%)
missing-error-handling108 (1.2%)
index-out-of-bounds97 (1.1%)
missing-functionality91 (1.0%)
feature-addition88 (1.0%)
missing-method-implementation79 (0.9%)
incomplete-implementation65 (0.7%)
missing-exception-handling57 (0.7%)
race-condition55 (0.6%)
Total tasks
32,079
Mean difficulty
5.80
Median difficulty
6.2
Avg patch lines
86.9
Avg patch files
4.22
Difficulty distribution SWE-rebench-V2
| Label breakdown | easy | medium | hard |
|---|---|---|---|
7319 / 13490 / 11268 |
7,319 | 13,490 | 11,268 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 32,077 | — | — | 6.2 | 5.80 | — | — |
Score bins
<=34,590 (14.3%)
3.1-55,972 (18.6%)
5.1-710,247 (31.9%)
7.1-86,982 (21.8%)
>84,286 (13.4%)
Languages
python7,222 (22.5%)
go6,147 (19.2%)
javascript4,371 (13.6%)
typescript3,989 (12.4%)
rust2,933 (9.1%)
java1,750 (5.5%)
php1,406 (4.4%)
kotlin857 (2.7%)
julia793 (2.5%)
elixir410 (1.3%)
scala399 (1.2%)
swift359 (1.1%)
dart251 (0.8%)
r159 (0.5%)
cpp158 (0.5%)
Area / tier
backend12,927 (40.3%)
library11,597 (36.2%)
frontend4,125 (12.9%)
cli3,419 (10.7%)
fullstack8 (0.0%)
framework1 (0.0%)
Top topics
swc407 (9.4%)
kubernetes356 (8.2%)
eslint297 (6.8%)
react268 (6.2%)
pandas250 (5.7%)
moto232 (5.3%)
serverless198 (4.5%)
statamic188 (4.3%)
cfn-lint162 (3.7%)
diktat158 (3.6%)
rust-analyzer157 (3.6%)
cobra153 (3.5%)
helm150 (3.4%)
pennylane135 (3.1%)
aiohttp133 (3.1%)
qiskit132 (3.0%)
swift-syntax117 (2.7%)
sveltekit117 (2.7%)
hypothesis114 (2.6%)
obsidian-linter111 (2.6%)
clap109 (2.5%)
laravel107 (2.5%)
knative103 (2.4%)
kiota101 (2.3%)
scalameta97 (2.2%)
Bug classes
incomplete-validation2,494 (22.4%)
missing-metadata-propagation1,525 (13.7%)
type-handling-inconsistency1,376 (12.3%)
missing-feature1,271 (11.4%)
missing-fallback1,124 (10.1%)
wrong-default1,060 (9.5%)
missing-validation363 (3.3%)
missing-state-propagation279 (2.5%)
incomplete-parsing223 (2.0%)
missing-implementation215 (1.9%)
false-positive163 (1.5%)
race-condition157 (1.4%)
missing-error-handling150 (1.3%)
index-out-of-bounds140 (1.3%)
missing-method128 (1.1%)
missing-default110 (1.0%)
missing-method-implementation104 (0.9%)
nil-pointer-dereference100 (0.9%)
missing-function-implementation88 (0.8%)
off-by-one85 (0.8%)
Total tasks
22,806
Mean difficulty
5.36
Median difficulty
5.8
Avg patch lines
61.8
Avg patch files
2.41
Difficulty distribution OpenSWE-filtered
| Label breakdown | easy | medium | hard |
|---|---|---|---|
6209 / 11129 / 5468 |
6,209 | 11,129 | 5,468 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 22,806 | — | — | 5.8 | 5.36 | — | — |
Score bins
<=34,214 (18.5%)
3.1-54,627 (20.3%)
5.1-78,497 (37.3%)
7.1-84,547 (19.9%)
>8921 (4.0%)
Languages
python22,686 (99.5%)
fortran37 (0.2%)
cpp25 (0.1%)
java9 (0.0%)
cython8 (0.0%)
typescript4 (0.0%)
c4 (0.0%)
javascript4 (0.0%)
solidity4 (0.0%)
rust3 (0.0%)
julia2 (0.0%)
django2 (0.0%)
go2 (0.0%)
elixir1 (0.0%)
markdown1 (0.0%)
Area / tier
backend11,616 (50.9%)
library8,096 (35.5%)
cli1,928 (8.5%)
frontend1,165 (5.1%)
fullstack1 (0.0%)
Top topics
django507 (14.4%)
pandas300 (8.5%)
pydantic275 (7.8%)
fastapi200 (5.7%)
click184 (5.2%)
numpy154 (4.4%)
xarray150 (4.3%)
argparse149 (4.2%)
pytorch145 (4.1%)
scikit-learn123 (3.5%)
scipy111 (3.2%)
flask108 (3.1%)
matplotlib107 (3.0%)
jax103 (2.9%)
qiskit103 (2.9%)
sqlalchemy101 (2.9%)
opentelemetry97 (2.8%)
home-assistant85 (2.4%)
ansible81 (2.3%)
jinja278 (2.2%)
requests77 (2.2%)
kubernetes74 (2.1%)
dask74 (2.1%)
asyncio72 (2.0%)
fonttools65 (1.8%)
Bug classes
incomplete-validation1,833 (21.2%)
type-handling-inconsistency1,268 (14.7%)
missing-metadata-propagation1,016 (11.8%)
missing-fallback993 (11.5%)
missing-feature895 (10.4%)
wrong-default815 (9.4%)
missing-validation318 (3.7%)
missing-implementation227 (2.6%)
missing-state-propagation193 (2.2%)
missing-error-handling164 (1.9%)
missing-default131 (1.5%)
missing-method113 (1.3%)
index-out-of-bounds99 (1.1%)
incomplete-implementation87 (1.0%)
incomplete-parsing86 (1.0%)
missing-method-implementation85 (1.0%)
missing-functionality81 (0.9%)
race-condition78 (0.9%)
feature-addition78 (0.9%)
missing-parameter-propagation70 (0.8%)
Total tasks
20,181
Mean difficulty
5.98
Median difficulty
6.3
Avg patch lines
321.4
Avg patch files
6.06
Difficulty distribution Scale-SWE
| Label breakdown | easy | medium | hard |
|---|---|---|---|
2314 / 12361 / 5506 |
2,314 | 12,361 | 5,506 |
| count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|
| 20,181 | — | — | 6.3 | 5.98 | — | — |
Score bins
<=3795 (3.9%)
3.1-53,808 (18.9%)
5.1-710,072 (49.9%)
7.1-85,506 (27.3%)
>80 (0.0%)
Languages
python19,284 (95.6%)
c176 (0.9%)
cpp152 (0.8%)
typescript72 (0.4%)
scala61 (0.3%)
java60 (0.3%)
javascript50 (0.2%)
yaml43 (0.2%)
rust42 (0.2%)
cmake36 (0.2%)
dart31 (0.2%)
csharp28 (0.1%)
kotlin24 (0.1%)
go23 (0.1%)
shell19 (0.1%)
Area / tier
library10,187 (50.5%)
backend7,988 (39.6%)
cli1,275 (6.3%)
frontend726 (3.6%)
fullstack4 (0.0%)
framework1 (0.0%)
Top topics
fonttools310 (8.0%)
click283 (7.3%)
globus-sdk213 (5.5%)
more-itertools207 (5.3%)
flatbuffers175 (4.5%)
beets164 (4.2%)
django164 (4.2%)
aioesphomeapi164 (4.2%)
pydantic159 (4.1%)
xsdata155 (4.0%)
geopandas148 (3.8%)
pymongo148 (3.8%)
jsonargparse146 (3.8%)
mesa144 (3.7%)
coloraide131 (3.4%)
pymarkdown125 (3.2%)
poetry-core125 (3.2%)
poetry124 (3.2%)
traitlets121 (3.1%)
ops119 (3.1%)
pytest118 (3.0%)
grizz115 (3.0%)
mcuboot112 (2.9%)
troposphere108 (2.8%)
latex2mathml107 (2.8%)
Bug classes
incomplete-validation1,472 (18.1%)
missing-feature1,429 (17.5%)
missing-fallback1,128 (13.8%)
type-handling-inconsistency920 (11.3%)
missing-metadata-propagation832 (10.2%)
wrong-default694 (8.5%)
missing-validation205 (2.5%)
missing-method203 (2.5%)
missing-functionality183 (2.2%)
missing-implementation145 (1.8%)
missing-error-handling128 (1.6%)
feature-addition115 (1.4%)
incomplete-parsing112 (1.4%)
performance-optimization101 (1.2%)
missing-state-propagation97 (1.2%)
missing-type-annotations90 (1.1%)
missing-parameter89 (1.1%)
missing-default79 (1.0%)
race-condition67 (0.8%)
missing-api-method60 (0.7%)