[FEAT] add device control

[RaSan147]

Add "allow-device-control" in user.json (False)
Toggle switch in webUI
On 1st toggle user needs to agree from commandline running terminal window
Then will verify,
Verify process:

1. Browser request for token
2. Server generate 2 token:
   i. One to use as url query so that browser can go to (probably use uuid.uuid4() library to generate a random token)
   ii. Other in the get request in above link (similarly)
3. Server send 1st token and generate a site with it and put the 2nd there
4. Browser request 127.0.0.1:port/?verifyDevice=token
5. Make the 4 request url Allow all origin header response in server
6. If Server gets request (that means same device, since 127.0.0.1 can only be used from same device), sends 2nd token, browser gets that
7. Browser sends the second token to server (same as step 4). Server approves both in browser and server and saves (that token) in user.json and localstorage. Since its in localstorage, unless malicious intent, it won't be shared
8. Also the request message like "raise volume" will be revarified (using step 4) with the token.

It won't hurt to make some extra requests rather than letting someone control your pc

This is also done in every login or verify user

[UzairURK]

@RaSan147,
Hope you are doing well.
Can I work on this issue, if so, assign it to me and give contribution guidelines.
Thanks

[RaSan147]

@UzairURK Just make sure to add soms good comments and file a PR

Modify the app_server.py to add accepting more url and query strings. Use pyrobox.py to check server documentation (readable enough)

Try to make the verification process in a different file if possible.

Thanks for your eagerness, don't worry if anything doesn't feel right to me, ill modify it.

[RaSan147]

@UzairURK Well what it needs most (and why i am postponing this) is because it needs the vast possibilities of what can be and should be controlled with chat or voice.
The server - Web-UI update will take me 2min to complete.

So I'd be glad if you simply make an text command input and do something based tools in a file (cli program will be enough and 1 input per command should be enough, I'll try to come up with similar inputs)

I'll merge it with the project ASAP

[UzairURK]

Thanks,
I'll keep updating you.

[RaSan147]

some examples:

1. increase/decrease volume (will change master volume of device)
2. pause music will pause media (follow media key, like the keys on laptop or new keyboards with Fn)
3. take Screenshot
4. paste (will paste clip board)
   etc

[RaSan147]

Thanks, I'll keep updating you.

Thanksss, also added CONTRIBUTION section in the readme

[UzairURK]

some examples:

1. increase/decrease volume (will change master volume of device)
2. pause music will pause media (follow media key, like the keys on laptop or new keyboards with Fn)
3. take Screenshot
4. paste (will paste clip board)
   etc

Alright

[UzairURK]

@RaSan147
have you used any Machine Learning Models? and I have some new ideas to be implemented in this?

[RaSan147]

@RaSan147 have you used any Machine Learning Models? and I have some new ideas to be implemented in this?

Not yet, thats on the end of the bucket list, because creating more and more personalized (not robotic) responses and its taking time (and my university admission running)
Once i have enough enough IO to charecterize the AI, I'll move on.

Please do share your idea. I'm all open😺

[RaSan147]

Also the other issue on having ML is intent management and data collecting.
Training ML to reply on "open google" -> "ok" and actually open "google" in the middle by storing the data, thats where I'm struggling.
Training IO is easy but the middle work is something I don't know😥

[UzairURK]

great! my study is still running! The idea was to respond by Video with somehow real gestures if a person talks to her, AI should respond like Asuna is talking, the tone of AI should vary based on the location of the user and there must be a caption option if necessary for someone. hope you get it. Otherwise, we will talk on video chat in a free slot.

[UzairURK]

Also the other issue on having ML is intent management and data collecting. Training ML to reply on "open google" -> "ok" and actually open "google" in the middle by storing the data, thats where I'm struggling. Training IO is easy but the middle work is something I don't know😥

together, we will work on it.

[RaSan147]

great! my study is still running! The idea was to respond by Video with somehow real gestures if a person talks to her, AI should respond like Asuna is talking, the tone of AI should vary based on the location of the user and there must be a caption option if necessary for someone. hope you get it. Otherwise, we will talk on video chat in a free slot.

I'm also struggling with voice output, as for you know voice training and output requires intense hardware and i tried on my gtx 1660, a simple sentence took 1min, so without MS speech sdk online, it may not be possible to make well customized voice. Since i don't earn and its open source, planning to use any free alternative

Also gestures will be added once i implement voice. Currently I'm more focused on server scaling and IO data generation 😵

[UzairURK]

@RaSan147
I have run RUN_ME.py server. should I run other python files to get started?
once tell me the exact instructions to run the web on localhost, I am able just to see run RUN_ME.py.

[UzairURK]

alright

[RaSan147]

@RaSan147 I have run RUN_ME.py server. should I run other python files to get startd?

Did you install all the requirements 1st?
If yes then
Runme should give you your IP and server port
ie http://127.0.0.1:45454
Copy and open it in browser. Viola you have Asuna in your PC

[RaSan147]

As for the caption, once i add voice, I'll move chat in front screen with transparency, so above we'll see Asuna, botton we can type and see msg

[UzairURK]

http://127.0.0.1:45454

it's working. thanks

[RaSan147]

And sorry in advanced, it uses regular expressions to catch input so it only knows around 30-40 types inputs
If you use my demo server, i can check the unknown inputs and add them to handler

[RaSan147]

@UzairURK just make a separate file with input and action + print output. I'll merge it then

[UzairURK]

Alright

…

On Fri, 24 Mar 2023, 03:38 RaSan, ***@***.***> wrote: @UzairURK <https://github.com/UzairURK> just make a separate file with input and action + print output. I'll merge it then — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATYQJ6MWA4MJNXGEKEYZIF3W5TGGFANCNFSM6AAAAAAUQUK3UQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[s-b-repo]

nice

[RaSan147]

@UzairURK any update bro

[UzairURK]

@RaSan147
bro I have created two files and in which I create the two token but it fails the verification due to device Id, can you tell me that device Id should be random token or fetch the actual device id from device?

[RaSan147]

@UzairURK you don't need to make the verification tool
Just make a file that will simply take CLI command and do that work. Then I can easily integrate it.
Take your time, Ramadan Mubarak 😇
Maybe soon I'll make a branch with the verification activated so that you can do your part easily.

[UzairURK]

@RaSan147
just to implement the feature which takes CLI command and does the work as we do by built-in commands on the window or just give me an example of a CMD command and do what kind of work?

[RaSan147]

some examples:

1. increase/decrease volume (will change master volume of device)
2. pause music will pause media (follow media key, like the keys on laptop or new keyboards with Fn)
3. take Screenshot
4. paste (will paste clip board)
   etc

ie:

import os

from tools import change_volume
# can be found in https://github.com/AndreMiras/pycaw/blob/develop/examples/audio_endpoint_volume_example.py

cmd = input()
if cmd == "raise volume":
    change_volume(20) # raise by 20%
elif cmd == "lower volume":
    change_volume(-20)
elif cmd== "open download folder":
    # check os and use file manager accordingly 
    os.system("explorer.exe %USERPROFILE%/Downloads")
elif .....

Something like this, as many as possible

[RaSan147]

@RaSan147 bro I have created two files and in which I create the two token but it fails the verification due to device Id, can you tell me that device Id should be random token or fetch the actual device id from device?

I'd be more than happy to explain everything to you on the process, but the issue is the server "pyrobox" i made and using now has very little documentation, so if you try to update the server code yourself, it'd cause more work (not questioning your skill or something) afterwards.
And if you planned to write in Django, I'll need to rewrite it again, so not much help.
I'm sorry. If you need I'll write the code for you (extremely busy now)😥

I've rewritten the steps (in 1st comment) with clear explanation

[UzairURK]

some examples:

1. increase/decrease volume (will change master volume of device)
2. pause music will pause media (follow media key, like the keys on laptop or new keyboards with Fn)
3. take Screenshot
4. paste (will paste clip board)
   etc

ie:

import os

from tools import change_volume
# can be found in https://github.com/AndreMiras/pycaw/blob/develop/examples/audio_endpoint_volume_example.py

cmd = input()
if cmd == "raise volume":
    change_volume(20) # raise by 20%
elif cmd == "lower volume":
    change_volume(-20)
elif cmd== "open download folder":
    # check os and use file manager accordingly 
    os.system("explorer.exe %USERPROFILE%/Downloads")
elif .....

Something like this, as many as possible

I have created this type of code, in which you command something It will perform an action, I was just conforming that is this complete task. You have told in beginning to create two tokens can you just explain it? I will write code on my own.
So, let me confrom two thing I have to do 1) CLI command and 2) two tokens generate and verify the device. Is it right?
Thanks

[RaSan147]

Yeah, if you want to colab, the complete task is the 1) CLI command

I already have made modules and functions for 2) 2 token generate and verify. So there's nothing to do in that part.

[UzairURK]

Yeah, if you want to colab, the complete task is the 1) CLI command

I already have made modules and functions for 2) 2 token generate and verify. So there's nothing to do in that part.

Alright.