conditional-detr-resnet-50 by Microsoft: Open-Weight Model

Runs On

What it takes to serve conditional-detr-resnet-50 (44M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

Precision	Weights	Memory needed	Cheapest setup	Per hour	Also fits
16-bit	0.1 GB	0.1 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
8-bit	0.0 GB	0.1 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
4-bit	0.0 GB	0.0 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By Microsoft, published under apache-2.0, revision 8f8795fb7c31.

Conditional DEtection TRansformer (DETR) model trained end-to-end on COCO 2017 object detection (118k annotated images). It was introduced in the paper Conditional DETR for Fast Training Convergence by Meng et al. and first released in this repository. The recently-developed DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings for localizing the four extremities and…

Read Microsoft's full model card

Conditional DETR model with ResNet-50 backbone

Conditional DEtection TRansformer (DETR) model trained end-to-end on COCO 2017 object detection (118k annotated images). It was introduced in the paper Conditional DETR for Fast Training Convergence by Meng et al. and first released in this repository.

Model description

The recently-developed DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings for localizing the four extremities and predicting the box, which increases the need for high-quality content embeddings and thus the training difficulty. Our approach, named conditional DETR, learns a conditional spatial query from the decoder embedding for decoder multi-head cross-attention. The benefit is that through the conditional spatial query, each cross-attention head is able to attend to a band containing a distinct region, e.g., one object extremity or a region inside the object box. This narrows down the spatial range for localizing the distinct regions for object classification and box regression, thus relaxing the dependence on the content embeddings and easing the training. Empirical results show that conditional DETR converges 6.7× faster for the backbones R50 and R101 and 10× faster for stronger backbones DC5-R50 and DC5-R101.

Intended uses & limitations

You can use the raw model for object detection. See the model hub to look for all available Conditional DETR models.

How to use

Here is how to use this model:

from transformers import AutoImageProcessor, ConditionalDetrForObjectDetection
import torch
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("microsoft/conditional-detr-resnet-50")
model = ConditionalDetrForObjectDetection.from_pretrained("microsoft/conditional-detr-resnet-50")

inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)

# convert outputs (bounding boxes and class logits) to COCO API
# let's only keep detections with score > 0.7
target_sizes = torch.tensor([image.size[::-1]])
results = processor.post_process_object_detection(outputs, target_sizes=target_sizes, threshold=0.7)[0]

for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
    box = [round(i, 2) for i in box.tolist()]
    print(
            f"Detected {model.config.id2label[label.item()]} with confidence "
            f"{round(score.item(), 3)} at location {box}"
    )

This should output:

Detected remote with confidence 0.833 at location [38.31, 72.1, 177.63, 118.45]
Detected cat with confidence 0.831 at location [9.2, 51.38, 321.13, 469.0]
Detected cat with confidence 0.804 at location [340.3, 16.85, 642.93, 370.95]

Currently, both the feature extractor and model support PyTorch.

Training data

The Conditional DETR model was trained on COCO 2017 object detection, a dataset consisting of 118k/5k annotated images for training/validation respectively.

BibTeX entry and citation info

@inproceedings{MengCFZLYS021,
  author    = {Depu Meng and
               Xiaokang Chen and
               Zejia Fan and
               Gang Zeng and
               Houqiang Li and
               Yuhui Yuan and
               Lei Sun and
               Jingdong Wang},
  title     = {Conditional {DETR} for Fast Training Convergence},
  booktitle = {2021 {IEEE/CVF} International Conference on Computer Vision, {ICCV}
               2021, Montreal, QC, Canada, October 10-17, 2021},
}

Configuration

Architecture: ConditionalDETRForObjectDetection
Context length (tokens): 1,024
Layers: 6
Stored precision: float32
Model type: conditional_detr

Identity and Version

Repository: microsoft/conditional-detr-resnet-50
Publisher: Microsoft
Task: Object detection
Modality: Image
Library: transformers
Parameters: 44M parameters
Languages: Not stated by the source
Revision: 8f8795fb7c319c7862d4f4cd699e76bb09cf2593
First published: 2022-09-09
Last updated: 2024-05-08

Files and Weights

6 files, 348.5 MB in total. The weights are 2 files totalling 348.5 MB in bin, safetensors.

Weights2 files · 348.5 MB

Configuration2 files · 4.7 KB

Documentation1 file · 4.7 KB

Repository1 file · 1.4 KB

Every file

File	Type	Size	SHA-256
model.safetensors	Weights	174.2 MB	e8553b4fa875
pytorch_model.bin	Weights	174.3 MB	2b589a660bb3
config.json	Configuration	4.4 KB	—
preprocessor_config.json	Configuration	301 B	—
README.md	Documentation	4.7 KB	—
.gitattributes	Repository	1.4 KB	—

License and Download

License: apache-2.0
Access: Open weights, no gate
Download size: 348.5 MB

Download from Microsoft

Released by Microsoft through its official repository on Hugging Face. Read the license.

Built From

Described by arXiv:2108.06152
Trained on (disclosed) coco

Memory Requirements

Precision	Weights in memory
As published	348.5 MB
16-bit	0.1 GB
8-bit	0.0 GB
4-bit	0.0 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Built on This Model

Derived fromfashion-object-detection

Questions About conditional-detr-resnet-50

How much GPU memory does conditional-detr-resnet-50 need?

About 0.1 GB at 16-bit and 0 GB at 4-bit: the weights (44M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run conditional-detr-resnet-50 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use conditional-detr-resnet-50 commercially?

Yes. conditional-detr-resnet-50 is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

What is conditional-detr-resnet-50's context length?

1,024 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Object detection

fashion-object-detection

Yainage90

This model is fine-tuned version of microsoft/conditional-detr-resnet-50. You can find details of model in this github repo -> fashion-visual-search And you can find fashion image feature extractor model -> yainage90/fashion-image-feature-extractor This model was trained using a combination of two datasets: modanet and fashionpedia The labels are ['bag', 'bottom', 'dress', 'hat', 'shoes', 'outer', 'top'] In the 96th epoch out of total of 100 epochs, the best score was achieved with mAP 0.7542. Therefore, it is believed that there is a little room for performance improvement.

Open weights mit 44M parameters 1,024 tokens transformers

View model

Model · Object detection

dab-detr-resnet-50

IDEA-Research

This is the model card of a transformers model that has been pushed on the Hub. Use the code below to get started with the model. This should output The DAB-DETR model was trained on COCO 2017 object detection, a dataset consisting of 118k/5k annotated images for training/validation respectively. Following Deformable DETR and Conditional DETR, we use 300 anchors as queries. We select 300 predicted boxes and labels with the largest classification logits for evaluation as well. We also use focal loss (Lin et al., 2020) with α = 0.25, γ = 2 for classification. The same loss terms are used in bipartite matching and final loss calculating, but with different coefficients. Classification loss…

Open weights apache-2.0 44M parameters transformers

View model

Model · Object detection

rtdetr_v2_r50vd

Peking University

The RT-DETRv2 model was proposed in RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer by Wenyu Lv, Yian Zhao, Qinyao Chang, Kui Huang, Guanzhong Wang, Yi Liu. RT-DETRv2 refines RT-DETR by introducing selective multi-scale feature extraction, a discrete sampling operator for broader deployment compatibility, and improved training strategies like dynamic data augmentation and scale-adaptive hyperparameters. These changes enhance flexibility and practicality while maintaining real-time performance. This model was contributed by @jadechoghari with the help of @cyrilvallez and @qubvel-hf This is RT-DETRv2 consistently outperforms its predecessor across all…

Open weights apache-2.0 43M parameters transformers

View model

Model · Object detection

rtdetr_r50vd_coco_o365

Peking University

However, we observe that the speed and accuracy of YOLOs are negatively affected by the NMS. Recently, end-to-end Transformer-based detectors (DETRs) have provided an alternative to eliminating NMS. Nevertheless, the high computational cost limits their practicality and hinders them from fully exploiting the advantage of excluding NMS. In this paper, we propose the Real-Time DEtection TRansformer (RT-DETR), the first real-time end-to-end object detector to our best knowledge that addresses the above dilemma. We build RT-DETR in two steps, drawing on the advanced DETR: first we focus on maintaining accuracy while improving speed, followed by maintaining speed while improving accuracy.…

Open weights apache-2.0 43M parameters transformers

View model

Model · Object detection

rtdetr_r50vd

Peking University

However, we observe that the speed and accuracy of YOLOs are negatively affected by the NMS. Recently, end-to-end Transformer-based detectors (DETRs) have provided an alternative to eliminating NMS. Nevertheless, the high computational cost limits their practicality and hinders them from fully exploiting the advantage of excluding NMS. In this paper, we propose the Real-Time DEtection TRansformer (RT-DETR), the first real-time end-to-end object detector to our best knowledge that addresses the above dilemma. We build RT-DETR in two steps, drawing on the advanced DETR: first we focus on maintaining accuracy while improving speed, followed by maintaining speed while improving accuracy.…

Open weights apache-2.0 43M parameters transformers

View model

Model · Object detection

detr-resnet-50

AI at Meta

DEtection TRansformer (DETR) model trained end-to-end on COCO 2017 object detection (118k annotated images). It was introduced in the paper End-to-End Object Detection with Transformers by Carion et al. and first released in this repository. Disclaimer: The team releasing DETR did not write a model card for this model so this model card has been written by the Hugging Face team. The DETR model is an encoder-decoder transformer with a convolutional backbone. Two heads are added on top of the decoder outputs in order to perform object detection: a linear layer for the class labels and a MLP (multi-layer perceptron) for the bounding boxes. The model uses so-called object queries to detect…

Open weights apache-2.0 42M parameters 1,024 tokens transformers

View model