[Question] Tracking COFFEE CUP in current demo

[Goon]

Hello there,

Firstly I must say what you guys have done is amazing!

I've tried the demo to track my coffee cup, but it seems like it can only detect my cup in some specific angles/perspective, doesn't it?

Another question is, can it recognizes 2 cups with the same shape but different image labels?
For example:

If it detect the cup with the heart label, we will show a 3D heart. Otherwise, we will show a 3D flower. Would it be possible?

Thanks,

[Bourry Xavier]

Hi @mrgoonie

Thank you for your feedbacks.
The cup detector is a neural network learning from a 3D renderer.
We use about 5 cups 3D models and we put random lighting (a simple phong lighting model a random number of pointlights and random material coefficients). But we don't use textures yet, that's why it can work badly with some textured cups. But it would be very doable since the input renderer is written in THREE.js.

We have also limited the perspectives of the cup. The typical use-case is like in this video: https://www.youtube.com/watch?v=9klHhWxZHoc : the AR device is looking to the cup from top, and the cup is seen vertically.

It would be perfectly doable to recognize cups with different labels, we would need to train a specific neural network. We only need the 3D model of the cup (in .OBJ, .OFF, .DAE or .GLTF or whatever 3D format compatible with THREE.JS) with the 2 textures.

Best,
Xavier

[Goon]

Thanks @xavierjs. That's great information.

Is the model teaching tool developed with THREE.js as well? I would love to use it in my project, can I discuss further with you in terms of pricing and feasibility via email or something? (I've tried to reach you guys via [e-mail] but there is no responses so far)

[Bourry Xavier]

Hi @mrgoonie

I am sorry if you did not receive an answer yet, we are not very numerous @jeeliz, I am just back from a week off and we are preparing the GTC Munich which occurs in a few days.
The trainer is not available in open-source, I explain some reasons in this post:

jeeliz/jeelizGlassesVTOWidget#2 (comment)

The subject of AR is still very hot for us, we are thinking about Training as a Service business model or stuffs like that. About pricing and feasibility we currently just provide development services based on our APIs (custom trained networks, 3D modelization for training, middleware atop libraries) on a time basis (with standard Eastern USA prices).

[Goon]

Hi @xavierjs

I do understand that the trainer tool is not in open-source, I'm not expecting to do the model training myself 😄

Let say that if I have a Coke can that needs to be trained, then how long it would take you to do it? And what is the price?

After you trained it, will it be a model data file in JSON and I only need to load it in with your API?
Then will I be able to know the position of detected area in the canvas, as well as the scale?
Something like this:

You can reached me at [e-mail] as well.

Looking forward to you reply.

Much thanks,

[Bourry Xavier]

Hi @mrgoonie

This is a raw estimation of the cost:

• $50 for the 3D model of the can (find an existing one, cleaning, converting, put the texture, do 2-3 texture versions)
• $300 for the neural network trainer script, neural network compression and tests
• $300 for the training

so around $650 for the trained neural network. It will be a JSON file that you will be able to use with this library like basic4.json

but currently the detection window is always square, so the bounding box of the can would be a square, not a rectangle.

The position and scale in pixels can be computed from detectState.positionScale (cf readme) by multiplying x and sx by the video width, y and sy by the video height.

Best,
Xavier

[Goon]

Hi @xavierjs

Thanks for your quick response. $650 sounds reasonable to me, square boundary is also good. But how long it would take you to do that?

Thanks,
Duy

[Bourry Xavier]

Hello,

It will take 3 weeks from signing the contract. What's your email address to send it ?

Best,
Xavier

[Goon]

@xavierjs

Wow I didn't know it would take that long, I don't have that much time, would it be doable within 10 days (including the weekends)?

I mentioned my email in the previous comment, it's [e-mail]

Thanks,
Duy

[Bourry Xavier]

The longer part is the training of the network, I planned 1 week (sometimes I have to retry, or modify the network for a better performance). I can do this in 10 days with a rush fee.

Best,
Xavier

[Goon]

@xavierjs please send me an email regarding the contract template and your full quotation with the rush fee as well. I'm looking forward to it. Thanks mate!

[Bourry Xavier]

Perfect, I will send it to you today.