civitai
Image Source - Twitter

CSAM allegations land AI image platform CIVITAI in hot water

If you are someone who is into AI image generation, you may have used, or atleast heard of the website CIVITAI. The website went from becoming a portal for downloading AI models for Stable Diffusion, to becoming a more expansive platform for users to share their AI image generations. However, it looks like this seemingly innocent platform has landed in trouble, with its cloud service provider OctoML, pulling out of their partnership, after 404Media, found CIVITAI guilty of featuring sensitive content.

According to 404Media’s report, an internal communication indicated that OctoML knew that users were creating sexually explicit material that included nonconsensual images of real people and pornographic depictions of children. Upon the release of this report, OctoML put out a statement that they will be adding in filters, preventing CIVITAI users from creating such explicit content on the platform. However, they later announced that they would be pulling out of their partnership with the accused, entirely.

In November, 404Media also found out that CIVITAI was running a bounties program, where it was rewarding users for creating convincing AI images of celebrities. Initially, these images were harmless, but soon enough, there was a trove of NSFW images of celebrities, which raised a huge red flag. In response, CIVITAI released a filter that prevented users from creating images of “certain celebrities”, but that was not really a full time solution.

OctoML, who uses Amazon Web Services’ servers announced that they are no longer interested in partering with CIVITAI, stating that “We have decided to terminate our business relationship with Civitai. This decision aligns with our commitment to ensuring the safe and responsible use of AI.”