Playing with the AI again

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • EthylTheAardvark
    Senior Member
    • Nov 2016
    • 169

    Playing with the AI again

    I've done a couple of posts where I've been playing with Stable Diffusion over on the balloons forums. Today I'm trying to get it to generate inflatable toy ideas! Here's the setup:

    I'm using the Automatic1111 repo on an i7 with an nvidia 4090. All these images take in the ballpark of 15-20 seconds to produce. Rather than the stable diffusion model. I'm using Chermamixl (https://civitai.com/models/120033/chimeramixl) which has been trained on various mythical creatures. My prompt is along the lines of "A highly detailed and intricate inflatable toy in the form of an anthropomorphic ..." with some text after that describing materials, pose and body features as I feel they need them. You get a pretty wide variety of images from one prompt.

    The AI seems to generally have a pretty hard time with putting tails in the right place, frequently generates quadrupedal forms even though I specified anthropomorphic (As well as some other text trying to force upright poses.) Using "doll" rather than "toy" does seem to have better results for getting upright poses if those are all you want. Some of the quadrupedal ones that were generated were also pretty good.

    Look at a couple thousand of these things and you'll start noticing similarities depending on your phrasing. A lot of anthro fox/vulpera pics seem to have a tuft in the center of their head, for example. Almost all of them do, in fact.

    One way to refine these would be to generate a few thousand images and then pick the ones that look the best to the user and retrain the AI on those. It'd take a solid weekend to generate the images and probably a while to retrain the AI with the hardware I have. I have some free time in December, though, so I might take a crack at that and see if I can improve the quality for this application.

    Here are some of the better images that were generated​
    Attached Files
  • EthylTheAardvark
    Senior Member
    • Nov 2016
    • 169

    #2
    Ooh another good one! This is a rare overall good body, tail attached where it's supposed to, interesting detail work on it.
    Attached Files

    Comment

    • O'Dragon
      Senior Member
      • Jan 2024
      • 298

      #3
      They look cool. Definitely way way better than what I got out of stable before. I got just some garbled mess with weird forms and tails and distorted faces 😄
      May I ask if you would share a prompt you used?
      By the way great GPU. Thanks to all of the Optimaziations my 3070 holds up quite well with its 8gig VRam but sometimes I think it struggles quite a bit 😄

      Do you have any new images?

      Comment

      • EthylTheAardvark
        Senior Member
        • Nov 2016
        • 169

        #4
        If you're using Automatic1111, you should be able to drop these into pnginfo to get the prompt. Let me know if it doesn't work on the ones I uploaded, maybe they get stripped when you upload them. I believe the PNGs have all the info required to generate that specific image, including the seed. You can do stuff like drop the image into intpainting/outpainting with the same seed or set it to random and you can change it up in areas that don't exist or mask out part of the image and re-generate it with a random seed. The standard models I'm using don't seem to do really great with that, but it works better than I expected it to. When I have some time free, I want to download an inpainting/outpainting-specific model and try one of those.

        The prompt for that quadruped one was: "A highly detailed and intricate inflatable doll in the shape of an anthropomorphic vulpera. She is beautiful. She is tall and slender. She has cute painted eyelashes. She is made of soft and shiny vinyl. Side profile." with a negative prompt of "Metallic, fat, chubby, big head, extra feet, extra tail, extra ears, hair". You still get some of that crap even with the extra prompt, but it happens way less often.

        The AI spat out this one most recently. I like a lot of what's going on there, but don't really care for the tail and would like to get it to generate the rest of the legs. I'm planning to drop it into the gimp to remove the background and tail and then see if I can inpaint/outpaint the image to see if I can improve the characteristics.

        The starting prompt for this one was "A picture of a inflatable doll in the shape of a beautiful sexy female vulpera. She is tall and thin. She is shy and coy. She is adorable. She is sexy. Her features are well defined. She is highly detailed. She is made of very soft, shiny, translucent vinyl." with a negative prompt of "Metallic, short, stubby, fat, thick neck, extra legs, extra feet, extra ears, extra tails, thick legs".

        Since that's one of the better results, I could try to tailor my prompt toward characteristics in this image to see if I can generate more like them that way. I really need to collect all the best images I generate into a training folder and retrain the model using them. That would help the AI learn what features I'm looking for, specifically.
        Attached Files

        Comment

        • O'Dragon
          Senior Member
          • Jan 2024
          • 298

          #5
          Thanks for your detailed answer. Yeah png info was the first thing I tried but it seems the metadata gets erased when uploading here.
          That would be really nice if you could share the other prompts as well but the two prompts you posted are already a great template. I was specifically interested in the negative prompts and which of the typical parts like „highly detailed, 4k“ etc. you used.

          Concerning the cropped legs, I asked myself if one could try an open pose controlnet for just the pose. No hands or face. There are one or two open pose models for SDXL.
          Or what I also found can work great for such problems is either experimenting with terms for perspective or framing or make a very rough childish drawing in paint or gimp and using img2img with high denoise like 0.6 - 0.8. That seems to give stable enough guidance to still generate freely but to put the subject completely into the frame.

          Yeah I also thought about retraining or creating a Lora. Given for what people aleeady have created models for I was quite surprised that until now no one seems to have done one for inflatables.
          I found a Lora for balloon figures for SDXL. I tried with different weights but unfortunately it didn’t really help to guide the models more into the direction I want it to go.

          Really interesting technology. I‘ll do some experiments now. If I got something usable I‘ll post it here.

          Comment

          • EthylTheAardvark
            Senior Member
            • Nov 2016
            • 169

            #6
            Ah, I see. It's too bad they get stripped. I've found that the Euler and Euler A samplers seem to work the best for these. Sometimes I get better results with other ones, but the Euler ones seem to be the most consistently good.

            I'd experimented with weights to get me where I want to go and some of those images used them. I think several of these images were generated by the same prompt with just one or two words substituted (Tiger or Panther instead of Vulpera, for example.) ChimeraXL seems to understand "Anthropomorphic" better than the standard model as well.

            The first fox there was generated with:

            A highly detailed and intricate inflatable toy in the form of an (anthropomorphic:2.4) female fox. She is made of (soft and shiny vinyl:1.2). She is tall and slender. She is beautiful. She has cute painted eyelashes. Her muscles are well defined. She is orange with a white belly. She has a well defined face. Full body pose. Turned at a slight angle

            Negative prompt: metallic, fat, chubby, plump, big head, extra legs, no arms, pinched joints, pinched knees. Extra tails. Hair. Writing.
            Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 2097747880, Size: 768x768, Model hash: b725a429b1, Model: chimeramiXL_v24, Version: v1.6.0

            The Quadruped Tiger was:

            A photograph of a highly detailed and intricate inflatable toy in the form of an (anthropomorphic:1.6) female tiger. She is made of soft and shiny vinyl. She is tall and slender. She is beautiful. She has cute painted eyelashes. Her muscles are well defined. (45 degree profile)
            Negative prompt: metallic, fat, chubby, plump, big head
            Steps: 20, Sampler: DPM++ 2M SDE Exponential, CFG scale: 7, Seed: 2159423368, Size: 768x768, Model hash: b725a429b1, Model: chimeramiXL_v24, Version: v1.6.0​

            I've found that "A photograph" seems to frequently work better than just describing the model.

            The anthro tiger right afterward was:

            A photograph of a highly detailed and intricate inflatable toy in the form of an (anthropomorphic:1.6) female tiger. She is made of soft and shiny vinyl. She is tall and slender. She is beautiful. She has cute painted eyelashes. Her muscles are well defined. (She is standing upright on two feet:2.1) (45 degree profile)
            Negative prompt: metallic, fat, chubby, plump, big head, extra legs, no arms, pinched joints
            Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 3662668266, Size: 768x768, Model hash: b725a429b1, Model: chimeramiXL_v24, Version: v1.6.0​

            The next fox over was:

            A photograph of a highly detailed and intricate inflatable toy in the form of an (anthropomorphic:1.6) female vulpera. She is made of (soft and shiny vinyl:1.2). She is tall and slender. She is beautiful. She has cute painted eyelashes. Her muscles are well defined.
            Negative prompt: metallic, fat, chubby, plump, big head, extra legs, no arms, pinched joints, pinched knees
            Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 1883472650, Size: 768x768, Model hash: b725a429b1, Model: chimeramiXL_v24, Version: v1.6.0​

            The panther was:

            A photograph of a highly detailed and intricate inflatable toy in the form of an (anthropomorphic:1.6) female panther. She is made of soft and shiny vinyl. She is tall and slender. She is beautiful. She has cute painted eyelashes. Her muscles are well defined. (She is standing upright on two feet:2.1) (45 degree profile)
            Negative prompt: metallic, fat, chubby, plump, big head, extra legs, no arms, pinched joints, pinched knees
            Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 3099842692, Size: 768x768, Model hash: b725a429b1, Model: chimeramiXL_v24, Version: v1.6.0​

            You can see I was playing with various weights there, but it rarely seems to make a huge difference, and can lead to unsettling hallucinations on the part of the AI if you crank them up too high.

            I rarely specify colors or specific details and let Chimera's imagination do a lot of the heavy lifting. One of the better balloon ones I did in my other post was just:

            A translucent red latex balloon shaped like a female vulpera. Her features are exquisitely detailed. She is smiling, shy, coy, beautiful
            Negative prompt: large head, small body
            Steps: 30, Sampler: Euler a, CFG scale: 7, Seed: 2888089285, Size: 528x528, Model hash: b725a429b1, Model: chimeramiXL_v24, Denoising strength: 0.7, Hires upscale: 2, Hires upscaler: Latent, Version: v1.6.0​

            Comment

            • O'Dragon
              Senior Member
              • Jan 2024
              • 298

              #7
              I experimented for a while yesterday and the results are great. These prompts work really well.
              i changed doll to toy which gave me laying figures.
              I like the results very much.
              The phrase „in the shape of“ seems to me gives the AI the right command because during experimenting I had switched model and forgot to change it back and even with the other model the results weren‘t so bad.
              I also tried to add in people who ride them but until now I haven’t found the best workflow.
              I tried various combinations of inpainting, img2img but it didn’t really work.
              But I found that I had used the wrong inpainting settings so lets see maybe I‘ll get it to work today.

              Comment

              • O'Dragon
                Senior Member
                • Jan 2024
                • 298

                #8
                So after a lot of experimentation, here are a few results I got:


                Dragon 01 - 04 were done in ComfyUI, don't know if you know it, it's another UI llike Auto1111.
                Needs more learning time to be able to use it but is much faster than Auto1111. Especially with your GPU you should be able to create a lot of images really fast with SDXL.

                As speaking of SDXL: I found in your details that you used 768x768. Was it intentional?. Because SDXL was trained mainly on 1024x1024 so you could start with a higher resolution than you normally would with 1.5 models. Then you don't need highres fix to get to 1024.
                I found when working with SDXL models the results get even better if I start with a 1024 resolution as a base.

                Drago 01 - 04:

                Positive: A highly detailed and intricate inflatable toy in the shape of a blue cartoon dragon. She is beautiful. She is majestic. She has cute painted eyelashes. She is made of soft and shiny vinyl. Side profile

                Negative: ugMetallic, fat, chubby, big head, extra feet, extra tail, extra ears, hair (noticed the typo "ugMetallic" later but left it in as I liked the results)

                Sampler: Euler a
                Steps: 20


                Dragon 05:

                A highly detailed and intricate inflatable toy in the shape of a blue cartoon dragon. She is beautiful. She is majestic. She has cute painted eyelashes. She is made of soft and shiny vinyl. Side profile
                Negative prompt: Metallic, fat, chubby, big head, extra feet, extra tail, extra ears, hair
                Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 366211339, Face restoration: CodeFormer, Size: 1024x1024, Model hash: c0550be157, Model: chimeramiXL_v47, Style Selector Enabled: True, Style Selector Randomize: False, Style Selector Style: base, Version: v1.7.0​
                Attached Files
                Last edited by O'Dragon; 21-01-2024, 12:22. Reason: Added Details about Image Generation

                Comment

                • EthylTheAardvark
                  Senior Member
                  • Nov 2016
                  • 169

                  #9
                  Looks like you're getting some excellent results! I chose the resolution I used as that's what a variety of sites suggested for the SD2 model I was playing around with initially. I'm not sure what resolution Chimera was trained on, so the one I'm using might not be optimum in one direction or the other. It's seemed to work reasonably well, so I haven't tinkered with it all that much.

                  Comment

                  • O'Dragon
                    Senior Member
                    • Jan 2024
                    • 298

                    #10
                    Thanks.

                    Ahh yes, I understand. Chimeraxl uses SDXL as the basis, the model of the next generation, version 3, so to speak. This was trained on 1024px images.
                    But I‘ll try with 768 when I‘ll use controlnet with it because the generation times increase terribly with 1024 resolution on my gpu 😕

                    Do you have something new? And do you still think about retraining?

                    Comment

                    • EthylTheAardvark
                      Senior Member
                      • Nov 2016
                      • 169

                      #11
                      I don't have anything new at the moment and haven't had time to look into retraining. I've been curious about getting the AI to project other views of the image, but this doesn't seem to be a thing it's at all good at. I've also been playing with one of the depth mapping plugins with an eye toward building a 3D model that I could potentially print. If it could do either of those things reliably well, I could just drop an image into blender, clean up the 3D model, print it and ship that print off to Hongyi as a reference. The workflow feels quite close but isn't there just yet.

                      Comment

                      Working...