Skip to content

Benchmarking VIZIO, Playmaker and Javascript on iOS

Benchmarking VIZIO, Playmaker and Javascript on iOS

Postby ivankio » 15 Sep 2012, 05:04

With concerns to garbage collection and action games on mobile platform, and as discussed in these topics:
http://forum.antares-universe.com/what-actions-and-lb%CA%B9s-cause-garbage-collect--t772.html
http://forum.antares-universe.com/huge-performance-spikes-t681.html#p2029
http://forum.antares-universe.com/performance-issue-regular-spike--t571.html#p2056
http://forum.unity3d.com/threads/137045-Garbage-Collection#12
I wanted to investigate it further. I did a benchmark seeking specifically for those performance spikes having in mind that Neodrop always suggest that we should use less variables.

info.jpg

As discussed and by the way I interpret the results of my tests, we have a real problem that are those long pauses that are happening too often which are specially noticeable on action games. I think the mobile gamer do has in common sense that occasionally
he may experience some hiccup in any application. But as we are experiencing, Antares Universe is producing them consistently and for too long. While there are examples (Mini Motor Racing, ShadowGun) of fluid games made on Unity.

Neodrop recommends that we use less variables and less often. Good guideline, but:
1. I guess a lot of us is using Antares Universe within a higher level of abstraction than the average programmer. "Bad practices" are bound to happen not only because we are doing unoptimized processes, but also because the higher level of abstraction allows us to be more human friendly and aim for processes less optimal. Some of this friendliness (like procedurally animated interface) should be possible with current mobile hardware. But Vizio seems to be causing an exceptional overload reducing our abstraction more than if we were to code in C# or JS.

2. Even if we were to comply with very strict use of variables, how close to no-code/no-interactivity would we need to go? As the tests suggest, we can delay the time between 2 spikes, but the spike will be consistently long (~150ms or 6 fps on an ipod touch 4).
An equivalent code in Javascript also produce GC, but somehow the spike do not always drop the application bellow 15fps (60 ms). I suppose that's how some Unity action games can still be run without constant (noticeable) hiccups.

3. Anyway, between the 2 versions of Antares tests, we saw virtually no difference even with the additional save and load of 50 variables every frame.

We have learned that the root of the problem with garbage collections is natural to Unity itself. But Antares is particularly producing them more noticeable.

So I wonder, Neodrop... would a compiler for Antares Universe be possible? I guess that the graph interpretation is already quite good and it would be too hard to improve it any further, so my guess would point to copy the project replacing graphs for scripts (as they feel so analog).
I, for one, would gladly pay for the compiler as a separate product or a professional edition. As I understand, Vizio is already very complete and useful, but when targeting mobile (what not everyone does), we kinda really need that extra optimization.
As I recall, some developers do worry a lot about performance loss when evaluating programming environment. This would take it out of question and you could use the claim of exact same performance as advertisement.

Maybe it was a silly suggestion, maybe I'm mislead about what mobile platforms can really do, but it seems to me enough evidence that Vizio do need some work on those spikes or it is not a good option for mobile action game development. Is it correct?
Attachments
results.jpg
Last edited by ivankio on 15 Sep 2012, 14:35, edited 1 time in total.
ivankio
 
Posts: 31
Joined: 14 Sep 2011, 21:35

Re: Benchmarking VIZIO, Playmaker and Javascript on iOS

Postby ivankio » 15 Sep 2012, 14:32

I've tweaked the app removing the touch and ToStrings parts when testing. I've also improved the data gathered to be more consistent (turning off wi-fi, leaving the device at the same place - temperature matters, and making the test much longer - 18000 frames). I'll post the screenshot for now, later I'll compile it with some more tests but I think it's already clear.
On a note, where it says 300, it means actually nothing since less than 3 frames went bellow the spike threshold (set to 20 fps here) in the javascript and C# tests.
Attachments
ipod long.PNG
ivankio
 
Posts: 31
Joined: 14 Sep 2011, 21:35

Re: Benchmarking VIZIO, Playmaker and Javascript on iOS

Postby holyjewsus » 16 Sep 2012, 00:05

not to muddle your benchmarks, but I ran something much more rudimentary and found it rather striking so I thought I would post it.

On the left side of this profiler screenshot is the output of 832 spheres running almost the same exact code you have running (converted to c#)

On the right the same spheres with the universe graphs running what I believe to be similar code. I noticed that as pointed out in other threads there is garbage for each call in the universe graphs, where as there isn't any in the c# calls.

c# seemed was hovering around 250fps while universe was at 60.

Image

Syntax:
Using csharp Syntax Highlighting
using UnityEngine;
using System.Collections;

public class jumparound : MonoBehaviour {
                 Transform actor;
                float posx;
                float posy;
        // Use this for initialization
        void Start () {
               
                posx = 0.0f;
                posy = 0.0f;
         actor = this.GetComponent<Transform>();

        }
       
        // Update is called once per frame
        void Update () {
                posx = Random.Range(-5,5);
                posy = Random.Range(-5,5);
                actor.localPosition = new  Vector3(posx,0,posy);

        }
}

 
Parsed in 0.007 seconds, using GeSHi 1.0.8.4
holyjewsus
 
Posts: 390
Joined: 08 Apr 2011, 00:58

Re: Benchmarking VIZIO, Playmaker and Javascript on iOS

Postby Neodrop » 16 Sep 2012, 15:08

As I say before, too many Universe graphs in one scene is a bad way. In my mobile apps I have no more 10-15 graphs per scene. It can be very big graphs, but a total nodes and variables count in it much less than in your benchmarks.
By architecting reasons, any get/set variable value producing a bit of garbage (I have decrease it a lot some time ago, but it still presented).
I will try to clear it more. But can't say how soon.

What about code compilation... I waте to implement it in Universe2. But, I think it will be a complitely new one Universe.
User avatar
Neodrop
Администратор
 
Posts: 1068
Joined: 15 Jan 2011, 13:18

Re: Benchmarking VIZIO, Playmaker and Javascript on iOS

Postby ivankio » 22 Sep 2012, 18:22

I'm glad the compilation speculation was not off. Dying to know more about Universe2. I recall it being mentioned once or twice on skype, but the interest really grew up now.

But first, back to the validity of the tests. The benchmark was indeed a fictional stress test, but I did imagine that actor (or graph count) could be an issue so I factored it further bellow. It's been a busy week but I'm back to compile other tests I did, and then I was hoping to talk about the future of the spikes issue.

-----

First note, I changed the info gathered to count how many times the frame rate would drop to critical intervals. I arbitrarily set the frame rate intervals I was interested in:
X >= 27 : The way it's expected to run smoothly (as a premise that 30fps is our usual frame rate for mobile games).
20 <= X < 27 : Just a slow down, no big deal.
14 <= X < 20 : Noticeable hiccup.
9 <= X < 14 : Hiccup evident, control probably lost.
4 <= X < 9 : Freeze evident, where was I?
X < 4 fps : Everything longer than a 0,25 second pause feel very frustrating. Almost the feeling we expect on loading events.

So, here are the results with the same code as explained at the first post:
IMG_1055.PNG


Then, reducing the actor count from 50 to 10 (didn't care to run on Playmaker).
IMG_1050.PNG

Note that:
1. although less graphs did reduce the number of spikes, the spike remained very strong and present.
2. the graph for this test is really, really simple (shown in the first post). An actual application would have a design somewhat more complex :!: .
3. there was virtually no difference between the 2 VIZIO tests as I would expect adding variable usage to the second.
4. the block count was also reduced with the number of graphs. Further testing ahead.

It's safe to point there's a drawback compromised in using VIZIO. Anyway the point is not to be nitpicking it, 7 hiccups on 10 minutes (18000 frames) of gameplay would border acceptance. But this test was as simple as I can imagine, more than do actually nothing each call. :!:I know there are designs where you process data mostly on triggers, but they are not all. I, for one, am thinking about action games like Asteroids (http://www.youtube.com/watch?v=7iAG0qObDbA). I would think that something as primitive as that should be viable. So I did some other graphs to further test VIZIO's spikes:

info2.jpg
A second run with device freshly restarted provided pretty much the same result - just in case.
So could I conclude:
1. variable use do not represent any special charge over regular blocks.
2. every run of vizio blocks will introduce garbage and the garbage collections it produces will be quite slow.

I understand this problem is this severe with some mobile devices, but, as previously discussed by others, the spikes also bothers on the desktop which has adjusted higher expectations. So, back to the spike issue future, I'm glad to hear about a compiler for Universe2. Will Universe2 be compatible with 1? Do you have a rough idea of it's first releases? If the answer to these questions are not very good news, would you, Neodrop, be willing to develop a compiler (even if very rudimentary which would just produce a script file from a graph file) for the current Universe on demand? Would this be a money issue?

As to the fellow Universe users, how are you coping with the spike issues?
A) Not having them. Those spikes are produced by bad programming methods. (please share some info)
B) The application/game has natural pauses or is more event based so it can GC pretty much seamlessly.
C)Target platform runs them sufficiently fast or user won't care.
D) Using Universe just for prototyping, plan to recode later.
E) Migration thoughts.


Finally, on a side note, is VIZIO just a early name and I should adopt Antares Universe definitively? Or would VIZIO be the name of a part of Universe? :mrgreen:
ivankio
 
Posts: 31
Joined: 14 Sep 2011, 21:35

Re: Benchmarking VIZIO, Playmaker and Javascript on iOS

Postby RichMakeGame » 22 Sep 2012, 20:57

for me, I'm making a fast action game with quite a few enemies and entities (and thus, vector operations)- so the spikes have been quite apparent. I don't think it's purely a matter of good programming methods- if you make a game over a certain size and complexity, I don't know how you avoid using a fair number of vector calculations (eg, get velocity, get position, vector3 create, etc). Sure it can be reduced, but logic demands a certain amount. I have had some success optimising my vector calculations, but I do feel a tiny bit 'cheated'- making optimisations where the actual code speed is not a problem, only GC- and in cases where unityscript would not create the garbage (or so I think?).

I'm leaning towards option D. I'm currently learning unityscript- i'm not enjoying it much, and the prospect of re-making my graphs in script isn't making me happy, but I have to focus on performance. The reality is I want to make a fairly complex but fast/smooth game and I can't have spikes every 30 seconds (or about 5 seconds, before my optimisations)

what I hope is that if I 'sketch out' logic in antares, it will be fairly quick to recreate in script- once I learn how to script that is, heh
-Rich
RichMakeGame
 
Posts: 49
Joined: 16 Feb 2012, 20:30


Return to General Discussion

Who is online

Registered users: No registered users