# distilbert-base-uncased vs SecBERT: Specs, License and GPU Cost Source: https://savrn.com/models/compare/distilbert-base-uncased-vs-secbert Markdown alternate of the page above; the site index is https://savrn.com/llms.txt --- | Field | [distilbert-base-uncased](https://savrn.com/models/distilbert-base-uncased) distilbert/distilbert-base-uncased | [SecBERT](https://savrn.com/models/secbert) jackaduma/SecBERT | | --- | --- | --- | | Publisher | DistilBERT community | Kun | | Task | Fill mask | Fill mask | | Modality | Text | Text | | Parameters, as reported | 67M parameters | 84M parameters | | Architecture | DistilBertForMaskedLM | BertForMaskedLM | | Library | transformers | transformers | | Context length | 512 tokens | 514 tokens | | Repository size | 1.5 GB | 673.3 MB | | Artifact formats | safetensors, pytorch, jax, rust, tf | safetensors, pytorch | | License | apache-2.0 | apache-2.0 | | Access | Open weights, no gate | Open weights, no gate | | Memory at 16-bit (weights and margin) | 0.2 GB | 0.2 GB | | Cheapest GPUs at 16-bit, per hour | 1x MI300X, $1.85 | 1x MI300X, $1.85 | | Memory at 4-bit (weights and margin) | 0 GB | 0.1 GB | | Cheapest GPUs at 4-bit, per hour | 1x MI300X, $1.85 | 1x MI300X, $1.85 | | Revision viewed | 12040accade4 | 7c603df5bc4c | | Downloads reported by the hub | 6.7M | 400.6k | | Last observed | 2026-10-23 | 2026-09-18 | An evaluation row appears only where at least two of these models report the same benchmark with the same stated configuration, metric, unit and setup. Different evaluators stay named in each cell. Values are shown as reported: no unit conversion, no ranking. ## SAVRN's Notes on distilbert-base-uncased At 4-bit the memory line rounds to zero. That settles the hardware question for distilbert-base-uncased: 67M parameters, 0.2 GB needed at 16-bit, 0.1 GB at 8-bit, so the cheapest listed setup, one MI300X with 192 GB at $1.85 an hour, is far more card than the job. The task is fill mask, uncased, so english and English read the same, distilled from BERT base as teacher on bookcorpus and wikipedia with no human labels. Apache 2.0 lets you ship it commercially, modify it and redistribute it, provided the license, copyright notices and any NOTICE file stay attached and significant changes are stated. Context is 512 tokens, which caps the span you mask in one pass. The 1.5 GB download across 12 files carries safetensors, PyTorch, JAX, Rust and TensorFlow; pick one. Check that a masked-language-model head, which is what DistilBertForMaskedLM ships, matches your pipeline. SAVRN Research, 2026-09-18 ## SAVRN's Notes on SecBERT Security text has its own vocabulary, and this model was built around that fact: a 52,000-entry wordpiece vocabulary the publisher calls secvocab, matched to a corpus of cyber security papers. The fill-mask head is the starting point, not the product; the publisher positions it as a base for NER, text classification and Q&A. Six layers and 84M parameters come to 0.2 GB at 16-bit, so the cheapest Index setup, one MI300X with 192 GB at $1.85 per hour on-demand, is a card to share across many fine-tunes, not dedicate to one. Apache 2.0 means a fine-tuned derivative can ship commercially, redistributed, with the notices kept and changes stated. Check the 514-token context first: documents longer than that get chunked. Then compare your text against the training sets, APTnotes, Stucco-Data and CASIE, and note the publisher also trained a SecRoBERTa version and last updated this one on 2023-06-26. SAVRN Research, 2026-09-18 ## Questions ### Which is larger, distilbert-base-uncased or SecBERT? SecBERT (84M parameters) is larger than distilbert-base-uncased (67M parameters), by the parameter counts their publishers report. ### Which is cheaper to run, distilbert-base-uncased or SecBERT? At 4-bit, distilbert-base-uncased fits on 1x MI300X from $1.85 an hour and SecBERT on 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists. ### Can I use distilbert-base-uncased commercially? Yes. distilbert-base-uncased is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors. ### Can I use SecBERT commercially? Yes. SecBERT is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors. ## Related Comparisons - [distilbert-base-uncased vs distilroberta-base](https://savrn.com/models/compare/distilbert-base-uncased-vs-distilroberta-base) - [bert-base-chinese vs distilbert-base-uncased](https://savrn.com/models/compare/bert-base-chinese-vs-distilbert-base-uncased)