pipette
ENEnglish

Product Costing in Data Pipelines

Li Peng, Jie You, Feng Shi, Yihang Wu, Jianxin Wang, Weiran Liu, Xinyu Peng, Junhua Wang, Lei Zhang, Lin Qu, Xiaohui Yu, Kui Ren, Jinfei Liu

PreprintDice ser un gran avanceUso en el mundo real

En palabras de los autores

Large data-intensive enterprises spend millions of dollars monthly running data pipelines. Apportioning this cost back to individual products, known as product costing, is a prerequisite for data pricing and cost recovery. However, this process is fundamentally more challenging than physical manufacturing costing due to the unique characteristics of data pipelines, including the non-rivalrous nature of data that complicates the costing formulation, the prevalence of cyclic dependencies that distort the accuracy of results, and the massive scale of pipelines that leads to computational intractability. In this paper, for the first time, we formalize the problem of product costing in data pipelines as a manufacturing cost of data products (MCDP) problem to overcome the non-rivalrous nature of data. To solve MCDP, we propose CostApp, a hybrid algorithm that achieves both exactness and scalability, even for cyclic and massive-scale pipelines. CostApp performs a dimensionality reduction by isolating cyclic dependencies via a feedback arc set (FAS). It aggregates costs onto a compact set of key vertices using a scalable iterative approach, transforming the intractable computational problem into a reduced linear system that can be solved exactly and efficiently by a basic matrix-based method. Experiments on both real-world data and synthetic data show that CostApp scales to process a production pipeline with over 28 million entities within an hour while preserving exactness, supporting pipelines up to 10^2 larger than the direct matrix-based baseline.

Resultado principalEl resumen no menciona limitaciones.

En palabras simples Resumen con IA

Las grandes empresas gastan millones mensualmente en ejecutar pipelines de datos. Asignar estos costos a productos individuales es más difícil que en la manufactura física. Los autores formalizan esto como el problema del costo de manufactura de productos de datos (MCDP). Proponen CostApp, un algoritmo híbrido que maneja ciclos y escala masiva manteniendo exactitud. CostApp procesó más de 28 millones de entidades en menos de una hora en experimentos.

Escrito por Kimi K2.5 usando solo el abstract de arriba. Jev verificó que cada oración esté respaldada por las oraciones del abstract que cita y eliminó las que no. Pasá el cursor por una oración para ver de dónde sale.

Apareció: viernes, 16 de octubre. arXiv. Preprint, todavía sin revisión por pares.

Comentario de los autores: 24 pages