[WIP][Rendering] Blendshapes (on behalf of Noa7/Noah7071) #2136

Eideren · 2024-01-31T20:35:25Z

@noa7 / @Noah7071 has issues opening pull requests, I'm opening this one on his behalf with his approval

PR Details

Adds blendshape/morphtarget support for models, see #339

Importing process: Use the preexisting FBX pipeline, add a new toggle to the import UI to include morph targets, store morph targets into a new collection inside of the model asset. - Eideren: No toggle but I doubt it would see much use anyway
Animation Binding: Importing animations would have to retrieve and include morph target keys as well. When those animations run, the associated morph target of the model being animated should be affected as well.
Runtime: ModelComponent would have a new collection of weights, which could be floats with values from 0 to 1 for example. Each of those weights relate to a morph target in the Model referenced. Those values would be controlled from editor or at runtime through C#.
Rendering: Applying those morph targets over the model when rendering based on the associated weights, after the skeleton blending pass.
Documentation
- Add a new sample for morph target, that sample contains a scene with a model and a small script. The script changes morph target blending at run time to show users how one would set it up.
- Add a page to the Animation documentation that explains what morph targets are and how they can be used

Related Issue

Fix #339

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My change requires a change to the documentation.
I have added tests to cover my changes.
All new and existing tests passed.
I have built and run the editor to try this change out. - Eideren: I'll try it out shortly

If @tebjan and/or @ykafia could take a look at this one too that would be great :)

ykafia · 2024-01-31T20:53:44Z

Will try to take some time to review some things on Thursday/tomorrow :)

Eideren

Thanks again for those changes @noa7 / @Noah7071
Pretty sure you can delete Stride.Core.Assets.Editor_5yobaqbr_wpftmp.csproj from your changes - that's a temporary file
There's a lot of fairly minor things to clean up still, tried to mark most of them for you.

Eideren · 2024-01-31T20:37:12Z

build/Stride.sln

@@ -332,7 +332,7 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Stride.NuGetResolver", "..\
 EndProject
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "NuGetResolver", "NuGetResolver", "{158087CF-AF74-44E9-AA20-A6AEB1E398A9}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Stride.Core.Presentation", "..\sources\presentation\Stride.Core.Presentation\Stride.Core.Presentation.csproj", "{0C63EF8B-26F9-4511-9FC5-7431DE9657D6}"


Best to roll this change back, afaict it is not necessary

It's actually correct but since it isn't related to the changes of this PR, let's revert it indeed. Otherwise it might conflict later with changes on the editor.

sources/engine/Stride.Engine/Rendering/ModelRenderProcessor.cs

sources/engine/Stride.Rendering/Rendering/BlendShape/TransformationBlendShape.sdsl

sources/engine/Stride.Rendering/Rendering/BlendShape/TransformationBlendShape.sdsl.cs

Eideren · 2024-01-31T23:20:36Z

sources/engine/Stride.Rendering/Rendering/Shape.cs

+                Position = new Vec4[value.Length];
+                for (int i = 0; i < value.Length; i++)
+                {
+                    Position[i] = value[i] * .01f;


What's this multiplier about ? Is that because of how fbx scales things ? Is there a way to apply that on import through the fbx importer ?

Yes. I too was also not sure about this. The weight data in .FBX ranges 0-100. I normalized to 0-1 as more intuitive ranging to developers. but can remove the multiplier and keep it as it is. Please let me know what you think!

Ah, okay, that makes sense. Can you move the scaling to the importer code, that way if we have other formats that support blend shapes in the future they won't have this fbx-specific scaler ?

Eideren · 2024-01-31T23:20:58Z

sources/engine/Stride.Rendering/Rendering/Shape.cs

+    [DataContract]
+    public class Vec4
+    {
+        public float x { get; set; }
+
+
+        public float y { get; set; }
+
+        public float z { get; set; }
+
+        public float w { get; set; }
+
+        public Vec4()
+        {
+
+        }
+
+        public static implicit operator Vector4(Vec4 v)
+        {
+            return new Vector4(v.x, v.y, v.z, v.w);
+        }
+
+        public static implicit operator Vec4(Vector4 v)
+        {
+            return new Vec4() { x = v.X, y = v.Y, z = v.Z, w = v.W };
+        }
+
+
+    }


This struct seems unnecessary ?

Yes. It's gone. Done!

Eideren · 2024-01-31T23:21:26Z

sources/engine/Stride.Shaders/Compiler/EffectCompilerCache.cs

+
+            goto label_force_compile;
+


Will have to remove that before merge, just something to keep in mind

Eideren · 2024-01-31T23:21:57Z

sources/tools/Stride.Importer.FBX/MeshConverter.cpp

@@ -655,7 +669,7 @@ public ref class MeshConverter
 			auto buffer = buildMesh->buffer;
 			auto vertexBufferBinding = VertexBufferBinding(GraphicsSerializerExtensions::ToSerializableVersion(gcnew BufferData(BufferFlags::VertexBuffer, buffer)), gcnew VertexDeclaration(vertexElements->ToArray()), buildMesh->polygonCount * 3, 0, 0);

-			auto drawData = gcnew MeshDraw();
+			//auto drawData = gcnew MeshDraw();


Small cleanup here and further below

Eideren · 2024-01-31T23:23:58Z

sources/tools/Stride.Importer.FBX/Stride.Importer.FBX.vcxproj

+    <ProjectReference Include="..\..\engine\Stride.Rendering\Stride.Rendering.csproj">
+      <Project>{ad4fdc24-b64d-4ed7-91aa-62c9eda12fa4}</Project>
+    </ProjectReference>


Not sure what's going on here, can you quickly go over why you had to remove the <Private>False</Private> here ?

I had to manually update reference in Importer .vcxproj to the project Stride.Redenring vcproj. Each time I update the Stride.Redenring vcproj, removing the reference and adding back in .vcxproj. It should be auto update but for some reason it was not. I will roll back so this goes to merge!

IXLLEGACYIXL · 2024-01-31T23:40:22Z

sources/engine/Stride.Rendering/Rendering/Mesh.cs

+                Dictionary<int, Vector3> mappings = new Dictionary<int, Vector3>();
+                foreach (var tup_id_vec in Draw.VCPOLYIN)
+                {
+                    if (!mappings.ContainsKey(tup_id_vec.Item2))


use the new TryAdd of the dictionary to avoid double lookup in the dictionary

noa7707 · 2024-02-01T08:33:28Z

Project("{F

Will try to take some time to review some things on Thursday/tomorrow

@ykafia Thanks Youness. Look forward to your feedback!

Eideren · 2024-02-01T09:39:57Z

sources/engine/Stride.Rendering/Rendering/BlendShape/TransformationBlendShape.sdsl

+    for(int i=0;i<MAX_MORPH_TARGETS;i++)
+    {
+        float4 morphedShape=BSHAPEDATA[i* MAX_VERTICES+ vID][0];
+        blendPos=+float4(morphedShape[0]+BasisKeyWeight,morphedShape[1], morphedShape[2] , pos[3])*morphedShape[3];


did you mean to write blendPos+=float4... instead of blendPos=+float4... ?

Yes, error on my part, fixed to standard a= a+b format in latest

tebjan · 2024-02-01T18:56:51Z

Thanks a lot for the work, I'll try to have a look at it next week.

Noah7071 · 2024-02-02T15:24:28Z

Thanks a lot for the work, I'll try to have a look at it next week.

Thanks @tebjan, Look forward to your feedback!

Kryptos-FR · 2024-02-04T08:50:42Z

sources/targets/Stride.Core.targets

-      <AnalyzersNotToRemove Include="@(Analyzer)" Condition="$([System.String]::Copy(%(FullPath)).Contains('Stride'))" />
+      <AnalyzersNotToRemove Include="@(Analyzer)" Condition="$(FullPath.Contains('Stride'))" />


Revert that. It looks like you had a conflict but resolved it the wrong way.

It would otherwise conflict with #2105

Right now the build is fairly fragile wrt those lines - let's wait until the PR is finalized and revert only then that way Noah can keep working on this.

Kryptos-FR · 2024-02-04T08:51:03Z

sources/targets/Stride.Core.targets

-      <AnalyzersNotToRemove Include="@(Analyzer)" Condition="$([System.String]::Copy(%(FullPath)).Contains('Generator'))" />
-      <AnalyzersNotToRemove Include="@(Analyzer)" Condition="$([System.String]::Copy(%(FullPath)).Contains('SourceGeneration'))" />
+      <AnalyzersNotToRemove Include="@(Analyzer)" Condition="$(FullPath.Contains('Generator'))" />
+      <AnalyzersNotToRemove Include="@(Analyzer)" Condition="$(FullPath.Contains('SourceGeneration'))" />


Kryptos-FR · 2024-02-04T08:51:10Z

sources/targets/Stride.Core.targets

      <AnalyzersToRemove Include="@(Analyzer)" Exclude="@(AnalyzersNotToRemove)" />
-      <Analyzer Remove="@(AnalyzersToRemove)"/>


SeleDreams · 2024-02-05T23:49:01Z

Thanks for your work.
I'm pretty excited about this development since blendshapes are a feature I waited for for years but lacked the rendering skills and codebase experience to implement.

For anime styled 3D games this will be essential as those rely fully on blendshapes for facial expressions.

noa7 added 5 commits January 30, 2024 07:38

BASE COMM

ed0e939

Updates on Shader and resource marshaling

a3f044a

Update- cleanup

8a3c13f

Updates- clean up

a679906

Updates

afe10c0

Eideren commented Jan 31, 2024

View reviewed changes

IXLLEGACYIXL reviewed Jan 31, 2024

View reviewed changes

noa7707 and others added 2 commits February 1, 2024 09:43

Updates as by Eidren & IXLLEGACYIXL on Github PR review

0a516fd

Update shader- remove commented test code

6649b09

Eideren commented Feb 1, 2024

View reviewed changes

Noah7071 added 4 commits February 1, 2024 11:22

Update: Clean up to effect class & shader

6ed75b6

Scale Factor moved to Importer

5db94d9

Removed extra class Vec4

c04c84d

Fix to update scale factor in impoter

e66c0c8

Eideren mentioned this pull request Feb 2, 2024

Blendshape Documentation stride3d/stride-docs#239

Draft

Kryptos-FR requested changes Feb 4, 2024

View reviewed changes

Noah7071 added 2 commits February 7, 2024 13:31

Normal fix on blendshapes

36c3daf

Normal fix

06c130b

Eideren marked this pull request as draft February 8, 2024 06:46

Noah7071 added 5 commits February 18, 2024 23:42

IMPORTER CKECIIN

1751b13

Importer

af97e33

Spell fix

96f878d

Updates to animation importer

58ead58

Animation Updates

f13c8f0

Eideren mentioned this pull request Mar 29, 2024

[Asset] Unified 3D asset importer (on behalf of Noa7/Noah7071) #2163

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][Rendering] Blendshapes (on behalf of Noa7/Noah7071) #2136

[WIP][Rendering] Blendshapes (on behalf of Noa7/Noah7071) #2136

Eideren commented Jan 31, 2024

ykafia commented Jan 31, 2024

Eideren left a comment

Eideren Jan 31, 2024

Kryptos-FR Feb 4, 2024

Eideren Jan 31, 2024 •

edited

Noah7071 Feb 1, 2024

Eideren Feb 1, 2024 •

edited

Noah7071 Feb 1, 2024

Eideren Jan 31, 2024 •

edited

Noah7071 Feb 1, 2024

Eideren Jan 31, 2024

Noah7071 Feb 1, 2024

Eideren Jan 31, 2024

Noah7071 Feb 1, 2024

Eideren Jan 31, 2024

Noah7071 Feb 1, 2024

IXLLEGACYIXL Jan 31, 2024

Noah7071 Feb 1, 2024

noa7707 commented Feb 1, 2024

Eideren Feb 1, 2024

Noah7071 Feb 1, 2024

tebjan commented Feb 1, 2024

Noah7071 commented Feb 2, 2024

Kryptos-FR Feb 4, 2024 •

edited

Eideren Feb 4, 2024

Kryptos-FR Feb 4, 2024

Kryptos-FR Feb 4, 2024

SeleDreams commented Feb 5, 2024

		<AnalyzersNotToRemove Include="@(Analyzer)" Condition="$([System.String]::Copy(%(FullPath)).Contains('Stride'))" />
		<AnalyzersNotToRemove Include="@(Analyzer)" Condition="$(FullPath.Contains('Stride'))" />

		<AnalyzersToRemove Include="@(Analyzer)" Exclude="@(AnalyzersNotToRemove)" />
		<Analyzer Remove="@(AnalyzersToRemove)"/>

[WIP][Rendering] Blendshapes (on behalf of Noa7/Noah7071) #2136

Are you sure you want to change the base?

[WIP][Rendering] Blendshapes (on behalf of Noa7/Noah7071) #2136

Conversation

Eideren commented Jan 31, 2024

PR Details

Related Issue

Types of changes

Checklist

ykafia commented Jan 31, 2024

Eideren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eideren Jan 31, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eideren Feb 1, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eideren Jan 31, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noa7707 commented Feb 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tebjan commented Feb 1, 2024

Noah7071 commented Feb 2, 2024

Kryptos-FR Feb 4, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SeleDreams commented Feb 5, 2024

Eideren Jan 31, 2024 •

edited

Eideren Feb 1, 2024 •

edited

Eideren Jan 31, 2024 •

edited

Kryptos-FR Feb 4, 2024 •

edited