Model Release Discussions 2
lllyasviel opened this issue · comments
Because the previous thread is out-of-date and many new previews are released, new thread starts here.
I will leave out most of my questions until I see the workflow-oriented assistant and supporting color scribbles sections being completed in the document.
Thanks to the Dorothy model, I'm fine with the program being called style2paints v5. This is very exciting.
Regarding functionality:
Will Dorothy model support color hints and exporting the color layer only (without lineart)?
How well can it turn a messy sketch/lineart into a completed lineart using the Alice model and prompting?
Can it do manga-style hatching and shading?
If you plan to allow people to draw the lineart directly in the program, the program should use something that isn't gradio. It should use a framework that better supports drawing tools for artists, like your original SEPA program.
Regarding ethics:
The final model release should have all of anything v3's weights replaced with standard stable diffusion. Using a variant of novelai is the most controversial part of the project due to its origin, so it would be best to release the model without its reliance. As you mentioned in the previous thread, I am glad to hear that this change will not affect the model greatly.
The model should also use the same ethical licenses as Stable Diffusion does, e.g the RAIL license.
Considering how beneficial this can be for artists, where it performs best under human-drawn lineart (and thus, human ideas), I see no problems releasing this to the public with the above suggestions.
Will the program be available in one language? I would like Style2paints5 to be available in many languages (International languages These include:
Englishman
Spanish
Arabian
Russian
French
Chinese). I am interested in Russian. If yes, what?
Will there be model distillation done to reduce inference times?
Is there any work being done on lineart -> lineart? Lineart would be easier to edit and would prove to be a good middle step before going straight to colorization (for the alice method, lineart -> lineart would provide greater control whilst still requiring much less detailed original input)
Will we ever find out if SEPA is simple or studio level.(https://mobile.twitter.com/lvminzhang/status/1392143022221975554).
amazing work keep it up
Update:
Just like style2paints V4, style2paints V5 will not officially support manga panels. However, just like style2paints V4, users may try unofficial ways to handle manga panels. Handling both illustrations and manga panels is too time-consuming for a small research group.
About Node Graph System:
We are very afraid of adding a node graph system.
To the best of our knowledges, adding a node graph system seems always bring bad luck to many different projects. Many projects die out quickly after adding a node system. I can tell a lot examples and I trust you can also name a few. This seems because that node graph has essential violations to fundamental HCI principles and conceptual model rules.
Many users of Blender have some levels of concerns to simplify the node system. They have also made some changes to directly put some buttons inside a right click menu to use, rather than asking users to build a node graph.
Nuke is another exception but they are not really only using node graphs. Their graph system is more like UML diagrams. Nevertheless, even Nuke is less intuitive than Adobe After Effects, which simply put buttons on the interface. See also the market share between Nuke and AE.
In conclusion, we will not add node system before we absolutely need it. We believe this decision helps us escape from the “curse of node graph”, which has already killed many many projects.
About Node Graph System:
We are very afraid of adding a node graph system.
To the best of our knowledges, adding a node graph system seems always bring bad luck to many different projects. Many projects die out quickly after adding a node system. I can tell a lot examples and I trust you can also name a few. This seems because that node graph has essential violations to fundamental HCI principles and conceptual model rules.
Many users of Blender have some levels of concerns to simplify the node system. They have also made some changes to directly put some buttons inside a right click menu to use, rather than asking users to build a node graph.
Nuke is another exception but they are not really only using node graphs. Their graph system is more like UML diagrams. Nevertheless, even Nuke is less intuitive than Adobe After Effects, which simply put buttons on the interface. See also the market share between Nuke and AE.
In conclusion, we will not add node system before we absolutely need it. We believe this decision helps us escape from the “curse of node graph”, which has already killed many many projects.
Thanks for clarifying.
Upd:
V5 Dorothy has potential to become a “clean” method that is free from copyright concerns and AI-generated labels, since it only blend color and minor texture to human drawn contents – all shapes are still drawn by human.
Nevertheless, we still need to take a look at different legal standards of different countries. We will let you know once we have any good news.
Will it be possible to run V5 on cloud GPU somehow, or local installation is the only way? This software looks very promising, just was wondering if it will be available for users like me with AMD or not very powerful laptops/PC's overall :)
Our first round of technical preview is now finished! We should not have any new posts for the next few weeks. We look forward to seeing you next time.
Thank you all!
Update:
The release of Style2Paints V5 will be delayed again. We need to make more thoughtful considerations about technology and the potential effects.
We are committed to presenting technology in a safe and beneficial way for all artists. We will let you know when we are ready. Thank you for your support and understanding!
If possible it would be great if v5 supported any models the user wanted and had access too. Making Alice and Dorothy the default and officially supported models is fine, but allowing users to mess around and test other models would be great.
If possible it would be great if v5 supported any models the user wanted and had access too. Making Alice and Dorothy the default and officially supported models is fine, but allowing users to mess around and test other models would be great.
That way the community could train the models to be used for manga.
We are committed to presenting technology in a safe and beneficial way for all artists. We will let you know when we are ready. Thank you for your support and understanding!
please, whatever you do, do not cause the quality of your excellent work to dip trying to please luddites that want your destruction.
PLEASE create a discord server as it will speed up response times for certain issues people are having. Thank You!
If there is concern about model release, could paper/technical report/code be released first? Information like training method, hyper parameter, and model architecture is of interest to many people.
Releasing information for academic purpose should be safe, as other companies, research team, and individuals have done.
UPD:
We are actively working on releasing technical materials. Please stay tuned.
UPD: We are actively working on releasing technical materials. Please stay tuned.
Looking forward to it!
If possible, please include details about the hand crafted weights/layers. Specifically, the ones mentioned below.
some layer weights are manually written by Lvmin, typically some special connection layers.
The paragraph on README.md wasn't clear about how the weight are chosen. Reading it seem to suggest some weights were not trained, but decided through other means, with human supervision and without actually training them through the usual gradient descend? I'm unsure if I'm reading it right. It'd be really, really nice to have clarification about how the mentioned manually written process works.
It seems that they already released everything in
There is absolutely no reason not to release the lineart model, unless you intend to monetize it.
There is absolutely no reason not to release the lineart model, unless you intend to monetize it.
Did you forget about other users training the model.
UPD: We are actively working on releasing technical materials. Please stay tuned.
I have to ask something real quick regarding the release of V5. When is it going to be released? And if not, are there any other issues that needed to be discussed regarding this topic?
Wow this update is amazing! As an artist who has tested out the earlier versions of Style2paints this seems like a massive leap in quality/usefulness!
Dorothy would be so helpful in coloring in the webtoons I’m working on. Alice could be very useful if it can be trained with an input character concept art sheet, so it can call a custom character consistently, like you did with “Rem” example. This would make it so a webtoon could be created even from a rough storyboard vs finalizing line art.
I guess that can be true for Dorothy too, where it knows which character is which based on the line art so it can use the same color palette consistently without human guidance.
For the legal issues, I think your caution is warranted as we’ve recently seen stable diffusion and github copilot come under suits from artists/coders/getty images, etc.
I wonder if a solution can be created where the liability is on the user of the tool, similar to Elevenlabs Voice AI https://beta.elevenlabs.io/terms. You can upload your own voice and have it create a clone of it. Similarly could Style2Paints allow artists to submit their own works to be trained on so it can be used by said artist. Maybe giving a license to Style2Paints of their copyrighted work so they can use the tool and Style2Paint can train on it.
Maybe in the future there can be a database that artists opt into so their art can be used for training or even get paid a small fee every time it is used.
Regarding ethics: The final model release should have all of anything v3's weights replaced with standard stable diffusion. Using a variant of novelai is the most controversial part of the project due to its origin, so it would be best to release the model without its reliance. As you mentioned in the previous thread, I am glad to hear that this change will not affect the model greatly. The model should also use the same ethical licenses as Stable Diffusion does, e.g the RAIL license. Considering how beneficial this can be for artists, where it performs best under human-drawn lineart (and thus, human ideas), I see no problems releasing this to the public with the above suggestions.
There should be a few good alternative anime models out there that are native trained and not inheriting from NovelAI running on RAIL license.
Regarding ethics: The final model release should have all of anything v3's weights replaced with standard stable diffusion. Using a variant of novelai is the most controversial part of the project due to its origin, so it would be best to release the model without its reliance. As you mentioned in the previous thread, I am glad to hear that this change will not affect the model greatly. The model should also use the same ethical licenses as Stable Diffusion does, e.g the RAIL license. Considering how beneficial this can be for artists, where it performs best under human-drawn lineart (and thus, human ideas), I see no problems releasing this to the public with the above suggestions.
There should be a few good alternative anime models out there that are native trained and not inheriting from NovelAI running on RAIL license.
The issue is that the best ones are inherited from novel ai. If only WD used SD 1.5 instead of 2.
Regarding ethics: The final model release should have all of anything v3's weights replaced with standard stable diffusion. Using a variant of novelai is the most controversial part of the project due to its origin, so it would be best to release the model without its reliance. As you mentioned in the previous thread, I am glad to hear that this change will not affect the model greatly. The model should also use the same ethical licenses as Stable Diffusion does, e.g the RAIL license. Considering how beneficial this can be for artists, where it performs best under human-drawn lineart (and thus, human ideas), I see no problems releasing this to the public with the above suggestions.
There should be a few good alternative anime models out there that are native trained and not inheriting from NovelAI running on RAIL license.
The issue is that the best ones are inherited from novel ai. If only WD used SD 1.5 instead of 2.
And that's where the issue lies. But there are new Anime models being released in droves, that finding a good Anime model that doesn't use Novel AI is actually going to be slightly hard due to the numbers. Like, there's a lot of them.
IMO just release it the same way controlnet was - as a general-purpose "extension" to existing models. You don't even need to provide sd 1.4 or 1.5, or novelai or wd or anything like that, just provide raw model states of your relevant layers and allow for re-attaching it as a diff
IMO just release it the same way controlnet was - as a general-purpose "extension" to existing models. You don't even need to provide sd 1.4 or 1.5, or novelai or wd or anything like that, just provide raw model states of your relevant layers and allow for re-attaching it as a diff
🤷 (if it works the same way controlnet stuff does)
S2P should remain as a stand alone desktop program, as it is not related to stable diffusion.
IMO just release it the same way controlnet was - as a general-purpose "extension" to existing models. You don't even need to provide sd 1.4 or 1.5, or novelai or wd or anything like that, just provide raw model states of your relevant layers and allow for re-attaching it as a diff
🤷 (if it works the same way controlnet stuff does)S2P should remain as a stand alone desktop program, as it is not related to stable diffusion.
quoted from V5_Preview:
The model is not trained from scratch. The initial weight comes from three sources: (1) some layers are initialized with borrowed weights from a wild mixture of many stable diffusion models, and we manually picked some best layers from many models [2], during many experiments. (2) some layers are initialized from scratch. (3) some layer weights are manually written by Lvmin, typically some special connection layers.
But fair enough. It would still be pretty awesome if this was released as-is with the model they put together. It seems extremely useful.
I know that previously your team had some doubts about nodal editing. Now with the advent of ComfyUI, a node-based editor based on stablediffusion, will this influence whether your team ends up developing a node-based editor or not?
I know that previously your team had some doubts about nodal editing. Now with the advent of ComfyUI, a node-based editor based on stablediffusion, will this influence whether your team ends up developing a node-based editor or not?
A s2p which can add lighting and textures, needs a node system in order to achieve a studio level product.
Chainer or enchanr have a really good gui.
The party is over! Nvidia, along with Shutterstock, Getty Images, and Adobe decided to release their own models. They claim to use ethical and legally sourced training dataset. This is a good reason for them to stomp down any open-source model and new start-up coming to AI party. The future of open-source is bleak on generative AI. Owner of massive data (read: massive corporations) will use artists emotion and empathy to stomp any emerging competition. Not a problem for them because they legally own their data.
The party is over! Nvidia, along with Shutterstock, Getty Images, and Adobe decided to release their own models. They claim to use ethical and legally sourced training dataset. This is a good reason for them to stomp down any open-source model and new start-up coming to AI party. The future of open-source is bleak on generative AI. Owner of massive data (read: massive corporations) will use artists emotion and empathy to stomp any emerging competition. Not a problem for them because they legally own their data.
S2P is over 5 years old, and only recently started using Stable Diffusion as a base.
S2P is over 5 years old, and only recently started using Stable Diffusion as a base.
Look who is suing Stability AI now, Getty Images, they now play the same game. The difference is going forward, every open-source image generation project will easily get sued by claiming their data is tainted. Unlike corporations that control or own massive amounts of data, open-source project is at a disadvantage of having to purchase the data to improve their software. So basically, they gated the improvement of open-source project. They can set a high price to ensure others can't compete with them.
S2P is over 5 years old, and only recently started using Stable Diffusion as a base.
Look who is suing Stability AI now, Getty Images, they now play the same game. The difference is going forward, every open-source image generation project will easily get sued by claiming their data is tainted. Unlike corporations that control or own massive amounts of data, open-source project is at a disadvantage of having to purchase the data to improve their software. So basically, they gated the improvement of open-source project. They can set a high price to ensure others can't compete with them.
They would not be bothered in any way by an anime girl colouring system.
S2P is over 5 years old, and only recently started using Stable Diffusion as a base.
Look who is suing Stability AI now, Getty Images, they now play the same game. The difference is going forward, every open-source image generation project will easily get sued by claiming their data is tainted. Unlike corporations that control or own massive amounts of data, open-source project is at a disadvantage of having to purchase the data to improve their software. So basically, they gated the improvement of open-source project. They can set a high price to ensure others can't compete with them.
They would not be bothered in any way by an anime girl colouring system.
This is literally a open source coloring program/extension(?) whose sole purpose is to create colored works from line art. Doubt that whatever the corpos are doing is kind of similar to what Illyasviel's doing right now.
Beside that point, was there any update from Illyasviel regarding this version of Style2Paints? It's been a month and a half since we last heard from her, and she's been busy with other extensions ever since then.
Ok guys, the line-art model was released on ControlNet v 1.1
Now we wait until the summer for an update
.