I'm just to ignorant to this in context of how the reality just works.
Downloading content on the web is what the web is for. Using the data for training is something lawers need to figure out.
open3dlab should spend the time and energy, as unfortunate it is, to sue them to get this resolved.
Ablog post about this is basically 0 information. The stories about soooo many bots now crawling the internet and creating a lot more traffic is going through news and hn for month.
Only tangentially related, but I've recently realized that unless you're some sort of business with a valid requirement of that, you can just block incoming traffic from all of AWS, DigitalOcean, Azure, etc.
I think the spark came from some other HN comment I saw? So thank you for that, whoever wrote that.
That alone is not a sufficient strategy of course, but you can get rid of a lot of bots by just refusing to talk to where they tend to live.
Of course, if everyone were to do that, they'd eventually pivot to residential proxies. But I don't see that happening.
I initially thought this was a good idea too until another comment pointed out you'll lose most business or personal customers in offices because they connect through VPN gateways running on those cloud services (especially Azure)
I would argue that more often than not it might not. Really depends on the specifics though.
A lot of stuff on the web actually isn't really relevant for all audiences everywhere I'd say. It just happened to be accessible by all, because academics back in the day dreamt of a better open world that didn't really manifest :(
Let me tell you, at the business I work at, some of our customers work at a company that uses a commercial VDI service for all workstations couldn’t access our service because the VDI egress was in GCP and we had an AWS WAF rule banning datacenter IPs.
Obviously, we removed the rule because it was doing nothing useful but plenty of harm. Real bots come from residential IPs.
And even if we run with that assumption, one might still be fine. It's a trade-off, but, depending on the target audience, a few lost sales might be worth savings in other aspects.
Same, I just checked and this week the vast majority of the bot traffic I get is random residential IPs that all pretend to be the exact same Chrome 155 on Linux.
Then you are not looking. Already have to switch to resi proxies from time to time when encountering yet another smartass who decided to play security theater and block datacenter ips on his shitpile he calls website, because "OUR USERS MUST BE GOOD BOYS WHO LOVE TO LIVE UNDER SURVILANCE AND ALLOW ISP TO TRACK THEM 24/7" (and before you say something like "others can track you too!", your ISP has the ability to knock on your door, they should be given as little information as possible, let them do metadata analysis on per-person basis if they really want it, but no 1-click mass surveillance shit).
If you are valuable, you will be be scrapped. Turning user hostile in a hopeless endeavor to "protect yourself" is not the solution.
I can see that you're having strong emotions about this, but that rant is just hard to read and serves primarily as venting and not really to convince anyone of anything.
Unless one zooms out and thinks "okay, these guys seem to react negatively if we do exactly that. We should do more of it", which is probably not what you want.
Nobody cares what you do, you are not smart enough. If you even entertain the idea of "let's just block datacenter ips!", all you accomplish by doing that is showcasing your incompetence and lose all respect from legitimate users who are good at tech. The scraper will just turn the resi proxy and will continue as is.
But maybe selecting for tech illiterate cattle as your userbase is what you want, then I guess more power to you. Don't be surprised by the scorn though.
I'm sympatic to open3dlab but I think the focus on adult-content is just trying to spark outcry for cheap.
It has been shown times and times against that removing adult content from the training source (at least for diffusion models), results in generally lower-quality output, even outside of adult-content, particularly in terms of representing human bodies.
would the implied intent (from the inclusion of adult content) of training models on open3dlab content, without asking them, render Roblox's actions _less_ antisocial???
The current AI Ethos is morality is just another obstacle on the way to their IPO. ...Lets just hope after they get rich, they stay away, and dont bother us with books, videos and conference speeches on entrepreneurship mentoring...
we've spent the entire 2010s with lots of talk whether creating a scraper for google or linkedin and packaging that data to sell is illegal just for the entire thing to turn around where large companies are basically burning small servers down with requests to do the exact same thing
Man if only there was such a thing as good manners, good behaviour, accepted practices from a community to not massively hammer servers. If only these websites had been up for a decade with very few problems, before a notoriously manipulative, lying, deceiving, abusive company that makes money off of child abusers came in and just went hurr durr free data in the WORST possible way instead of just politely asking for a dump.
I know it is an unpopular opinion, but if you do not want your content to be scraped, you have to put it behind payment walls and copyright it. In Germany we have decisions from the highest courts that deemed websracping perfectly legal (it was in the context of a competitor scraping airline ticket fares).
maybe stupid question but how is it possible to not be protected against supercrawlers like this, when you own large amounts of data, specially if all requests are coming from the same server ? Aren't cyberprotection services like CloudFlare work against this kind of "attacks" ?
Yeah, I can imagine it being a problem if it was from different IP addresses, but surely putting a limit on requests from a single IP would be a straightforward fix? Although maybe until it happens, you don't think about it.
> The crawler did not respect our robots.txt, (which it did retrieve), ignored our Terms of Service, (which forbid scraping), and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only.
I mean, it's not the size of the download, it's the unauthorised scraping of community assets strictly against terms and ignoring robots.txt by an American corporation. Resource of a kind that used to get kids threatened with spending most of their life in prison.
But I agree that this kind of theft by US corporations is no longer news.
(Yes, it's theft: they were aware they may not do it and according to the article attempted to get around further restrictions. I mean it's literally even theft of the $3 you mention — you don't have to be in possession the $3 to illegally deprive its owner of it under common theft statutes)
Of course not. But there’s no indication whatsoever that any license has been violated. Downloading a file from a free access server cannot violate license terms.
Fear that something will happen is not the same thing as evidence it has happened.
Who actually cares? Isn't the whole premise of the AI revolution just gorging ourselves with other people's data so we can have AGI? I was under the impression that we have accepted this sacrifice with OAI and friends leading the way.
Downloading content on the web is what the web is for. Using the data for training is something lawers need to figure out.
open3dlab should spend the time and energy, as unfortunate it is, to sue them to get this resolved.
Ablog post about this is basically 0 information. The stories about soooo many bots now crawling the internet and creating a lot more traffic is going through news and hn for month.
I think the spark came from some other HN comment I saw? So thank you for that, whoever wrote that.
That alone is not a sufficient strategy of course, but you can get rid of a lot of bots by just refusing to talk to where they tend to live.
Of course, if everyone were to do that, they'd eventually pivot to residential proxies. But I don't see that happening.
https://news.ycombinator.com/item?id=49491791
None of these measures alone are a full solution, but if you stack a bunch of cheese slices, eventually, no holes line up anymore.
I would argue that more often than not it might not. Really depends on the specifics though.
A lot of stuff on the web actually isn't really relevant for all audiences everywhere I'd say. It just happened to be accessible by all, because academics back in the day dreamt of a better open world that didn't really manifest :(
Let me tell you, at the business I work at, some of our customers work at a company that uses a commercial VDI service for all workstations couldn’t access our service because the VDI egress was in GCP and we had an AWS WAF rule banning datacenter IPs.
Obviously, we removed the rule because it was doing nothing useful but plenty of harm. Real bots come from residential IPs.
And even if we run with that assumption, one might still be fine. It's a trade-off, but, depending on the target audience, a few lost sales might be worth savings in other aspects.
Then you are not looking. Already have to switch to resi proxies from time to time when encountering yet another smartass who decided to play security theater and block datacenter ips on his shitpile he calls website, because "OUR USERS MUST BE GOOD BOYS WHO LOVE TO LIVE UNDER SURVILANCE AND ALLOW ISP TO TRACK THEM 24/7" (and before you say something like "others can track you too!", your ISP has the ability to knock on your door, they should be given as little information as possible, let them do metadata analysis on per-person basis if they really want it, but no 1-click mass surveillance shit).
If you are valuable, you will be be scrapped. Turning user hostile in a hopeless endeavor to "protect yourself" is not the solution.
Unless one zooms out and thinks "okay, these guys seem to react negatively if we do exactly that. We should do more of it", which is probably not what you want.
But maybe selecting for tech illiterate cattle as your userbase is what you want, then I guess more power to you. Don't be surprised by the scorn though.
Maybe I'm an idiot and it is a bad idea. But also maybe not?
Isn't it good if I "showcase my incompetence and lose all respect"? Wouldn't that be beneficial to people that disagree with these ideas?
Blocking "scrapping [sic]" is also not security theater, you misunderstand the intent of GP.
It has been shown times and times against that removing adult content from the training source (at least for diffusion models), results in generally lower-quality output, even outside of adult-content, particularly in terms of representing human bodies.
What is the actual news here, can anyone explain ????
we've spent the entire 2010s with lots of talk whether creating a scraper for google or linkedin and packaging that data to sell is illegal just for the entire thing to turn around where large companies are basically burning small servers down with requests to do the exact same thing
> The crawler did not respect our robots.txt, (which it did retrieve), ignored our Terms of Service, (which forbid scraping), and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only.
But I agree that this kind of theft by US corporations is no longer news.
(Yes, it's theft: they were aware they may not do it and according to the article attempted to get around further restrictions. I mean it's literally even theft of the $3 you mention — you don't have to be in possession the $3 to illegally deprive its owner of it under common theft statutes)
Fear that something will happen is not the same thing as evidence it has happened.
I guess we'll never know until you learn to read anything other than sheet music.
Sure it can. If you're downloading it for commercial purposes (clearly the case) you're violating the license.
Which is different than using the data 1 to 1.