Can the initial Resource Moving be optimized further?
Validark opened this issue · comments
This is the code in question. I've considered coroutines, doing two repeat until loops, one for the stuff that should go to ReplicatedStorage and one for the stuff that shouldn't.
Coroutines slowed down my speeds in my last test, because it involves a lot of function calls, and currently it only uses the GetChildren() function (which is written in C++), aside from one or two GetFolder and GetLocalFolder calls for the top folder the Modules need to be parented to. The coroutine solution is much slower because it involves way more function calls.
The current implementation minimizes the number of checks that need to be done for each object, but is there any way to remove these checks entirely? The aforementioned method of two repeat until loops could work. I haven't explored it extensively but preliminary tests have yielded little change. See next comment for more details.
local Repository, ServerRepository, ServerStuff -- Repository folders
local Boundaries = {} -- This is a system for keeping track of which items should be stored in ServerStorage (vs ReplicatedStorage)
local Count, BoundaryCount = 0, 0
local NumDescendants, CurrentBoundary = 1, 1
local LowerBoundary, SetsEnabled
Modules = {ModuleRepository}
repeat -- Most efficient way of iterating over every descendant of the Module Repository
Count = Count + 1
local Child = Modules[Count]
local Name = Child.Name
local ClassName = Child.ClassName
local GrandChildren = Child:GetChildren()
local NumGrandChildren = #GrandChildren
if SetsEnabled then
if not LowerBoundary and Count > Boundaries[CurrentBoundary] then
LowerBoundary = true
elseif LowerBoundary and Count > Boundaries[CurrentBoundary + 1] then
CurrentBoundary = CurrentBoundary + 2
local Boundary = Boundaries[CurrentBoundary]
if Boundary then
LowerBoundary = Count > Boundary
else
SetsEnabled = false
LowerBoundary = false
end
end
end
local Server = LowerBoundary or Name:lower():find("server")
if NumGrandChildren ~= 0 then
if Server then
SetsEnabled = true
Boundaries[BoundaryCount + 1] = NumDescendants
BoundaryCount = BoundaryCount + 2
Boundaries[BoundaryCount] = NumDescendants + NumGrandChildren
end
for a = 1, NumGrandChildren do
Modules[NumDescendants + a] = GrandChildren[a]
end
NumDescendants = NumDescendants + NumGrandChildren
end
if ClassName == "ModuleScript" then
if Server then
Modules[Name] = Child
if not ServerRepository then
ServerRepository = GetLocalFolder("Modules")
end
Child.Parent = ServerRepository
else
if not Repository then
Repository = GetFolder("Modules")
end
Child.Parent = Repository
if not Modules[Name] then
Modules[Name] = Child
end
end
elseif ClassName ~= "Folder" and Child.Parent.ClassName == "Folder" then
if not ServerStuff then
ServerStuff = GetLocalFolder("Server", ServerScriptService)
end
Child.Parent = ServerStuff
end
Modules[Count] = nil
until Count == NumDescendants
ModuleRepository:Destroy()Testing two repeat until loops:
local Repository, ServerRepository, ServerStuff -- Repository folders
local Count, NumDescendants = 0, 1
Modules = {ModuleRepository}
local ServerThings = {}
local ServerThingsCount = 0
repeat -- Most efficient way of iterating over every descendant of the Module Repository
Count = Count + 1
local Child = Modules[Count]
local Name = Child.Name
local ClassName = Child.ClassName
local GrandChildren = Child:GetChildren()
local NumGrandChildren = #GrandChildren
if NumGrandChildren ~= 0 then
for a = 1, NumGrandChildren do
Modules[NumDescendants + a] = GrandChildren[a]
end
NumDescendants = NumDescendants + NumGrandChildren
end
local Server = Name:lower():find("server")
if Server then
ServerThingsCount = ServerThingsCount + 1
ServerThings[ServerThingsCount] = Child
else
if ClassName == "ModuleScript" then
if not Repository then
Repository = GetFolder("Modules")
end
Child.Parent = Repository
if not Modules[Name] then
Modules[Name] = Child
end
elseif ClassName ~= "Folder" and Child.Parent.ClassName == "Folder" then
if not ServerStuff then
ServerStuff = GetLocalFolder("Server", ServerScriptService)
end
Child.Parent = ServerStuff
end
end
Modules[Count] = nil
until Count == NumDescendants
Count = 0
repeat
Count = Count + 1
local Child = ServerThings[Count]
local Name = Child.Name
local ClassName = Child.ClassName
local GrandChildren = Child:GetChildren()
local NumGrandChildren = #GrandChildren
if NumGrandChildren ~= 0 then
for a = 1, NumGrandChildren do
ServerThings[ServerThingsCount + a] = GrandChildren[a]
end
ServerThingsCount = ServerThingsCount + NumGrandChildren
end
if ClassName == "ModuleScript" then
Modules[Name] = Child
if not ServerRepository then
ServerRepository = GetLocalFolder("Modules")
end
Child.Parent = ServerRepository
end
until Count == ServerThingsCount
ModuleRepository:Destroy()Times:
Original
0.0012824535369873
0.0013079643249512
0.0011956691741943
0.0013306140899658
0.0015277862548828
Modified, two repeat untils
0.0023126602172852
0.0012509822845459
0.0012736320495605
0.0013413429260254
0.00126051902771
0.0012862682342529
0.0013382434844971
This is just an initial test. I will work more on this another day, but I'm kind of tired and unable to see areas for improvement. I am just posting this to keep a record and to receive suggestions if anyone has any.
Will probably need to create a testing workbench that can repeat the task a few 100 times to best see results once the best version of the modifications suggested have been best implemented
@Narrev should we consider allowing the user to have the modules in ReplicatedStorage from the get-go?
The thought has crossed my mind, we could just completely remove the movement functionality
Pros:
- The user would have complete understanding and control of which Modules are replicated and which aren't
Cons:
- It would take away from the whole "unified codebase" thing
- The user would need to navigate two repositories for their game instead of one
We should note that Modules on Github should have some kind of Replicatable tag for whether it is server-side or not (for installing modules with the plug-in). It should then be placed in either the ServerStorage repository or ReplicatedStorage repository accordingly, if we separate the codebase.
Hi. This is already closed, but I want to make a note.
As the saying goes, "Premature optimization is the root of all evil"
Please note that 0.001 seconds is a very small time for something that happens once and is bootstrapping the whole game.
I would highly prioritize readability and reliability over performance in the bootstrapper. Keeping it simple is much more important than this minor optimization. Nevermore's old bootstrapper as of now is simply to allow lookup of resources by name instead of by path, that is, allow code to be reorganized easily without having issues with submodules.
My two cents.
I can agree with @Quenty here. Further optimization cannot reward enough performance to compensate the fact that the code base becomes increasingly more difficult to read, understand and edit.
I wasn't going for 0.001 seconds here. In fact, I was hoping to explore the possibility of halving setup time. Because the changes I made had a negligible impact (noise had more impact than the changes), I closed this issue because I concluded that we are already loading resources as fast as possible until Roblox adds a GetDescendants function.
This is not premature optimization, this is optimizing the set-up code of a 6-year-old plus bootstrapper that entire games run on.
My bootstrapper works 100% the way I want it to, every single time. It is completely reliable, and if you have a problem with it, we have a dedicated issues tab at the top of this page.
I have decided to leave the Resource initialization the way it is. I don't think it is too complicated or unreadable, and if you have either of those issues, you can keep your version however you want it. I would also point out that this module is not part of "the codebase". It is what organizes and runs the codebase, and does not need to be understood by every individual that uses it. All users need to know is the API.
I've experimented with various solutions to handling RobloxLua's inability to yield during metamethod functions (LuaJIT solved this problem), and one of the bad solutions was to not yield for the top-level folders, and replace functions with smart tables. This did not work for devSparkle. Why?
devSparkle has given me the perspective that Roblox servers are slow. Sometimes really slow. Because when clients used to not yield for the initial folder creation from the server, he would report to me that every so often someone's Nevermore client would break because the folder creation hadn't occurred yet on the server, or at least hadn't been replicated yet.
We have since switched over to having the client yield for every resource (for the very reason aforementioned), but the point is that having sluggish code where it counts isn't some kind of virtue. The module-moving code is a derivative of Utility.CallOnDescendants if you don't understand it.
I am certainly not going to change Utility.CallOnDescendants or the current module-moving-setup-code just so you can feel like you can read it easier.
If you have any questions concerning how it works, I would be happy to go through how the code works with you. In fact, I think I might release a YouTube video just for you two explaining line by line, why each choice was made and exactly what happens where for every case.
This module can do everything your old one can do and much more, all the while doing it faster and with arguably more intelligent error messages. If you can make it faster, submit a pull request. If you have an issue or feature request, I would be more than happy to write a fix or reasonable feature for this module that I constantly maintain.
Otherwise, you are just criticizing a job well-done, and I don't want to hear it.
@Narrev while I agree with you here, I'd just like to correct that what I meant by readability was that we should maintain a certain degree as to allow for quick onboarding for other contributors to the framework.
@devSparkle If you have a specific suggestion or complaint, I can address it. Otherwise, all I can respond with is:
I'll keep that in mind.