am i reading correctly that the compression is just a relational records? i.e. o... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		iririririr 7 hours ago \| parent \| context \| favorite \| on: We gave terabytes of CI logs to an LLM am i reading correctly that the compression is just a relational records? i.e. omit the pr title, just point to it?

		help

aluzzardi 7 hours ago [–]

There are 2 layers of compression:

- ZSTD (actual data compression)

- De-duplication (i.e. what you're saying)

Although AFAIK it's not "just point to it" but rather storing sorted data and being able to say "the next 2M rows have the same PR Title"

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact