pipette
ENEnglish

Generative Text Modelling and Language-Guided Visual Attention for Deciphering Palimpsests

Robin Hollifeldt, Tianru Zhang, Jessica Str\"om, Ekta Vats

Preprint

En palabras de los autores

Recovering obscured text in palimpsests, i.e. the manuscripts where the original writing (under-text) was erased and overwritten for reuse, is vital for preserving lost information. Due to the nature of the material, such a task can be highly challenging and time-consuming. This paper presents an approach towards automated palimpsest handwritten text modeling for under-text recovery. We investigate generative models, including Generative Adversarial Networks (GANs) and diffusion models, and design a custom denoising diffusion model that learns structured noise patterns to reconstruct the under-text. The recovered text is subsequently processed by a fine-tuned, attention-based handwritten text recognition network. Furthermore, we investigate a language-guided recognition framework that leverages textual class descriptions as informative priors, enabling the text recognition model to distinguish meaningful features from noise. This is achieved by grounding task-specific descriptions, phrased in words or sentences, through visual attention maps generated by pretrained vision-language models like CLIP (Contrastive Language-Image Pre-training). To facilitate future research and benchmarking, we release new datasets for synthetic palimpsests (currently scarce), CLIP-compatible text descriptions, and pre-trained models with code.

Resultado principalEl resumen no menciona limitaciones.

Apareció: martes, 20 de octubre. arXiv. Preprint, todavía sin revisión por pares.

Comentario de los autores: This is a pre-MIT Press publication version, accepted for publication at TACL