Ask HN: Estimation of copyright material used by LLM
1. Is it true that LLMs / AI Companies have used copyrighted material for training?
2. Is it possible to estimate how much of copyrighted material has been used?
1. Is it true that LLMs / AI Companies have used copyrighted material for training?
2. Is it possible to estimate how much of copyrighted material has been used?
I think what you're looking for is not "copyrighted material" but material that's both 1) used without permission and 2) outside the scope of fair use.
There's no easy answer there, hence New York Times v. OpenAI.
There is an easy answer, it's just obfuscated by powerful people who are benefiting from it an obscene amount, and supported by hoards of addled and thoroughly addicted enthusiasts.
I think sticking a straw in Zlib or AA or LibGen or whatever it is, and drinking until it makes gurgling slurping noises as it hoovers up the dregs at the bottom of the barrel, is far, far removed from “fair use”.
1. Yes, but it's hard to prove. There are active lawsuits. Some of it has been under "fair use" but at the billion dollar scale, you have to really ask whether it's fair. Also anecdotally, an author friend lamented that her publisher sold the legal rights to use it... it was all perfectly legal but many authors do not agree to this.
2. This is harder as a lot of them don't disclose training sets.
pretty much everything newer than ~70 years old on the internet is copyrighted, because copywright occurs automatically when you create something (in the US at least). So the answer to #1 is yes.
1. Yes 2. No