GenoArmoryBenchmark Read paper
Benchmark paperarXiv 2025

GenoArmory: A Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Models

One benchmark for attacking and defending DNA foundation models.

Read the paper View code Leaderboard Model checkpoints arXiv

Authors

Haozheng Luo1,*Chenghao Qiu2,*Yimin Wang3Shang Wu1Jiahao Yu1Zhenyu Pan1Weian Mao4Haoyang Fang5Hao Xu6Han Liu1Binghui Wang7Yan Chen1

Equal contribution.

Affiliations

  • 1Northwestern University
  • 2Texas A&M University
  • 3University of Michigan
  • 4Massachusetts Institute of Technology
  • 5Carnegie Mellon University
  • 6Brigham and Women's Hospital & Harvard Medical School
  • 7Illinois Institute of Technology
01

Motivation

Genomic foundation models are now used for promoter detection, variant-effect prediction, and sequence generation. Nobody had measured, on common ground, how easily they can be pushed into wrong answers.

DNA is a structured, high-dimensional, low-redundancy input. A change of a few nucleotides can matter biologically, and earlier work had already shown that DNABERT-2 and the Nucleotide Transformer can be misled by nucleotide- and codon-level substitutions. Yet existing GFM benchmarks such as GUE and GenBench score accuracy only. They say nothing about robustness.

Results that did exist were hard to compare: models tokenize DNA differently (k-mers, BPE, single nucleotides), use different architectures, and are evaluated on different datasets. GenoArmory fixes the models, datasets, hyperparameters, and metrics, so attacks and defenses can be compared directly.

Figure 1: overview of the benchmark. The attack side lists gradient-based (PGD) and non-gradient (TextFooler, BertAttack, FIMBA) methods, adversarial sequence lengths from 30 to 1000, and Hyena- and transformer-based targets. The defense side lists FP16 and W8A8 or OutEffHop quantization, model sizes, the FreeLB, adversarial training and ADFAR defenses, and the promoter, transcription factor and epigenetic mark datasets.
Figure 1. What GenoArmory varies on the attack side and on the defense side. Original figure from the paper. Read the introduction ↗

The attack setting

Given a sequence \(X = [x_1,\dots,x_n]\) with \(x_i \in \{\mathrm{A},\mathrm{T},\mathrm{C},\mathrm{G}\}\), a fine-tuned model \(f\), and the true label \(y\), the attacker looks for a nearby sequence that the model misclassifies. Here \(d\) measures the perturbation and \(\epsilon\) is the budget. Every model in the benchmark is evaluated as a sequence classifier, including GenomeOcean, a generative model that is fine-tuned for classification.

\[ f(X') \neq y \quad \text{subject to} \quad d(X, X') \le \epsilon. \]
02

Methodology

View figure

Fix the models, data, and metrics; then plug in any attack or defense and read off comparable numbers.

GenoArmory packages five GFMs fine-tuned on 26 DNA classification datasets, four attacks, three defenses, and a visualization tool, behind one Python and command-line interface built on PyTorch and Hugging Face Transformers. Because only one attack had been designed for genomics (FIMBA), the others are adapted from language and vision, and each is run from its official implementation.

The attack outputs are also released as an artifact repository and distilled into GenoAdv, a dataset of adversarial examples for adversarial training.

What the benchmark contains
ComponentIncludedNotes
Target modelsDNABERT-2 · NT · NT2 · GenomeOcean · HyenaDNAFour transformers and one Hyena model; GenomeOcean is the generative one
AttacksBertAttack · TextFooler · PGD · FIMBAToken substitution, gradient-based, and the genome-specific FIMBA; AutoAttack and a universal attack in the appendix
DefensesAdversarial training · FreeLB · ADFARAT uses TextFooler examples for augmentation
Data26 datasets · 5 tasks · 4 speciesFrom GUE; sequences of 70 to 1,000 bp
New resourcesGenoAdv · attack artifacts · visualizationAdversarial examples from BertAttack, TextFooler, and FIMBA across models

Benchmark contents as described in Sections 3 and 4 of the paper.

Figure 2: the GenoArmory framework. An attacker applies BertAttack, TextFooler, PGD, FIMBA and other attacks to the GFMs to produce a wrong prediction; adversarial training, ADFAR, FreeLB and other defenses aim to keep the original prediction. GenoArmory also provides a visualization tool and the GenoAdv dataset.
Figure 2. The GenoArmory framework: attacks, defenses, visualization, and the GenoAdv dataset. Framework figure from the paper.
  1. 01 /

    Attack success rate

    An attack is scored by the relative drop in accuracy it causes, from clean inputs to adversarial ones. Higher means the model is more vulnerable:

    \[ \mathrm{ASR} = \frac{A_{\mathrm{clean}} - A_{\mathrm{adv}}}{A_{\mathrm{clean}}} \times 100\%. \]
  2. 02 /

    Defense success rate

    A defense is scored by how much accuracy it keeps under attack. Higher is better, and with no defense the score works out to \(100\% - \mathrm{ASR}\):

    \[ \mathrm{DSR} = \left(1 - \frac{A_{\mathrm{def}} - A_{\mathrm{adv}}}{A_{\mathrm{def}}}\right) \times 100\%. \]
  3. 03 /

    One interface for every method

    An attack, a defense, or a lookup of stored attack artifacts is a single call with a parameter file, so a new method can be evaluated under the same settings as the baselines:

    from GenoArmory import GenoArmory
gen = GenoArmory(model=model, tokenizer=tokenizer)
gen.attack(attack_method="pgd", **kwargs)
gen.defense(defense_method="freelb", **kwargs)
gen.get_attack_metadata(method="TextFooler", model_name="dnabert")
Notation

\(A_{\mathrm{clean}}\) is the accuracy of the fine-tuned model on clean test inputs and \(A_{\mathrm{adv}}\) its accuracy on adversarial inputs. \(A_{\mathrm{def}}\) is the accuracy of the defended model on adversarial inputs. In the result tables, the rows labelled N/A equal \(100\% - \mathrm{ASR}\) from the attack table to within rounding. See Appendix I.2 of the paper.

Installing and running the toolkit

The package is on PyPI as genoarmory and can also be installed from source. The paper's appendix shows the same operations from the command line: attack, defense, visualize, and read, each taking a model path and a JSON parameter file. Fine-tuned checkpoints are in the Hugging Face collection. See Appendix F.

pip install genoarmory

python GenoArmory.py --model_path magicslabnu/GERM attack \
  --method pgd --params_file scripts/PGD/pgd_dnabert.json
03

Leaderboards

5 models · 4 attacks · 4 defenses · 26 datasets

Every result in the benchmark, ranked. Each board can be narrowed to one task or one dataset.

Attack leaderboardDefense leaderboardModel robustness

Attack leaderboard

Which attack breaks genomic foundation models most often. Attack success rate on each model; higher means a stronger attack.

Attack leaderboard

Defense leaderboard

Which defense keeps the most accuracy under attack. Defense success rate per attack, or per model; higher is better. The no-defense baseline is listed last for reference.

Columns
Defense leaderboard

Model robustness

Which model is hardest to attack. Attack success rate against each model; lower means more robust.

Model robustness

Propose a new attack, defense, or model

04

Selected results

Attacks, defenses, quantization & GenoAdv

Average ASR over five GFMs higher means more vulnerable

37.10%99.00%

Text attacks beat the genomic one

FIMBA → BertAttack

TextFooler ASR on DNABERT-2 lower is better

92.37%19.90%

Quantization blunts the attack

Full precision → W8A8

Attacks borrowed from NLP already work

BertAttack and TextFooler, both designed for text, reach average success rates of 99.00% and 94.72%. FIMBA, the only attack designed for genomic models, averages 37.10%, the lowest of the four; the paper attributes part of this to limits in the released FIMBA code and its original evaluation setup. Rank 1 marks the model an attack hurts most.

Attack success rate · Table 1 · rank in brackets, 1 = highest ASR
AttackTransformer-basedHyena-basedAvg
DNABERT-2NTNT2GenomeOceanHyenaDNA
BertAttack96.23%(5)99.87%(1)99.56%(4)99.57%(3)99.75%(2)99.00%
TextFooler92.37%(4)96.69%(2)96.56%(3)99.54%(1)88.45%(5)94.72%
PGD38.28%(2)38.23%(3)34.41%(5)36.57%(4)47.94%(1)39.09%
FIMBA39.94%(2)37.66%(3)36.50%(4)41.06%(1)30.35%(5)37.10%
Model average66.71%(3.25)68.11%(2.25)66.76%(4.00)69.19%(2.25)66.62%(3.25)

Averages over all datasets and three seeds; the paper reports variances of at most 2%. In the bottom row the bracket is the model's average rank across the four attacks. GenomeOcean, the generative model, has the highest average ASR, and it shares the most-attacked average rank with NT. NT2 sits furthest from the top, which the paper links to BPE tokenization: a perturbed subword can keep part of the original token. HyenaDNA has the lowest average ASR but a middling rank, because it is the most-attacked model under PGD and the least-attacked under TextFooler and FIMBA.

Full table & discussion
Figure 3: heat map of ranks from 1 to 5 for DNABERT-2, NT2, NT, GenomeOcean and HyenaDNA on each of the 26 datasets, grouped into epigenetic marks, promoter detection, and human and mouse transcription factor prediction.
Figure 3. Ranks of the five models by ASR on each of the 26 datasets; OG is GenomeOcean. Original figure from the paper. Source ↗

No defense wins everywhere

ADFAR gives the largest gains against BertAttack and TextFooler, for example lifting NT from 3.31% to 67.07% DSR under TextFooler, yet it is the weakest of the three defenses against PGD on every model. Against PGD, FreeLB is best on three of five models, which the paper attributes to its smoothing of the adversarial loss during training. Plain adversarial training trails ADFAR on both text attacks, with one exception: HyenaDNA under TextFooler, where it is the best defense.

AT is adversarial training with TextFooler-augmented data. Even the best defense leaves BertAttack largely effective: the highest DSR against it in Table 2 is 21.84%, ADFAR on DNABERT-2. Every number is in the defense leaderboard ↑ · Table 2 and Section 4.2 ↗

Training on GenoAdv instead of TextFooler samples

The same adversarial-training recipe, run once with TextFooler examples and once with GenoAdv, scores higher with GenoAdv in 12 of the 15 model–attack pairs. The exceptions are NT2 and HyenaDNA under PGD and HyenaDNA under TextFooler. The paper summarizes the gain as a 34.71% improvement over TextFooler-only training.

Compare the two adversarial-training rows of the defense leaderboard, by model and for one attack at a time. The paper's Table 3 caption names the metric ASR, but its values coincide with the DSR rows of Table 2, so the leaderboard lists them as DSR. In the appendix, GenoAdv training also scores above adversarial training under AutoAttack and a universal attack in 9 of 10 cases; the exception is GenomeOcean under the universal attack, 55.54 against 55.55. Open the defense leaderboard ↑ · Table 3 and Section 4.4 ↗

Quantization makes the attacks weaker

With 8-bit weights and activations (W8A8), attack success drops on both models tested, sharply for DNABERT-2 and slightly for NT. The paper's explanation is that quantization creates flat regions in the loss landscape. With OutEffHop, an outlier-removal quantization method, the ASR is slightly higher than with vanilla quantization in all four cases, which the paper reads as outlier removal eliminating part of that flatness. TextFooler suffers more than BertAttack, plausibly because it relies on precise word-importance scores.

Attacks on quantized models · Table 4 · ASR, lower is better
QuantizationBertAttackTextFooler
DNABERT-2NTDNABERT-2NT
None (full precision)96.2399.8792.3798.23
W8A8, vanilla59.4699.3719.9066.57
W8A8, OutEffHop64.7199.4221.3468.53

ASR in %. Quantization follows the setup of GERM and SparQ. The full-precision TextFooler value for NT is 98.23 in this table and 96.69 in Table 1; both are reproduced as published. See Table 4 and Section 4.5 ↗

Where the attacks change the sequence

The visualization tool counts how often an attack modifies each BPE token position. For BertAttack on DNABERT-2 in the three 300 bp promoter-detection tasks, edits occur along the whole sequence but become more frequent in the later positions, roughly tokens 40 to 60, before dropping off at the end. The paper reads positions that attacks modify often as the parts the classifier relies on, and its abstract reports that the attacks frequently target biologically significant regions.

Figure 4: three histograms of modification frequency by token position for prom_300_tata, prom_300_notata and prom_300_all under BertAttack on DNABERT-2. Modifications occur at all positions, rise toward positions 40 to 60, and fall to near zero after about position 64.
Figure 4. How often BertAttack modifies each BPE token position, DNABERT-2, promoter detection (300 bp). Source ↗
05

Evaluation context

Each model is fine-tuned from its official checkpoint for sequence classification on the 26 GUE datasets, following DNABERT-2, and the fine-tuned models are the targets. Attacks and defenses run from their official repositories. Every experiment is repeated with three random seeds, and all reported variances are at most 2%.

Models
DNABERT-2 117M · NT 2.5B · NT2 100M · GenomeOcean 100M · HyenaDNA 4M
Tasks
Epigenetic marks (10 datasets) · promoter detection (3) · core promoter detection (3) · transcription factor prediction, human (5) and mouse (5)
Fine-tuning
AdamW · learning rate \(3\times 10^{-5}\) · 4 epochs · batch 64 · max length 256 · warmup 0.05
Quantization
W8A8, vanilla and OutEffHop
Compute
4 NVIDIA H100 GPUs · 80GB each

Several limits follow from the setup. The benchmark covers classification only; GenomeOcean is evaluated as a fine-tuned classifier rather than as a generator, so the comparison between classification and generative models is a comparison of how each fine-tunes. Quantization is tested on two models and two attacks. Attack strength depends on the released implementations: the paper flags FIMBA's code as a likely reason for its low numbers, and the appendix's AutoAttack and universal-attack results report identical ASR across all five models.

The work is dual-use, and the paper treats it that way. The authors disclosed their findings and artifacts to the DNABERT-2, NT, HyenaDNA, and GenomeOcean teams before release. Code and data are for non-commercial research. Raw adversarial outputs, GenoAdv included, are not published openly; access goes through the authors, with requests reviewed by healthcare cybersecurity professionals on the team. Clinical or diagnostic use is explicitly discouraged.

Ethics and dual-use statement ↗
06

Citation

BibTeX
Download 
@misc{luo2025genoarmory,
  title         = {GenoArmory: A Unified Evaluation Framework for
                   Adversarial Attacks on Genomic Foundation Models},
  author        = {Haozheng Luo and Chenghao Qiu and Yimin Wang and
                   Shang Wu and Jiahao Yu and Zhenyu Pan and Weian Mao and
                   Haoyang Fang and Hao Xu and Han Liu and Binghui Wang and
                   Yan Chen},
  year          = {2025},
  eprint        = {2505.10983},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2505.10983}
}