Shifu: The New Infrastructure of LLM Era

Challenges and Difficulties in Implementing Data Collection, Processing, and Device Control with AI

Large language models such as ChatGPT can provide a lot of help throughout the entire chain from access to application development. However, there is still much work to be done before ChatGPT can truly converse with these devices. Each device uses different protocols and has different rules, making it difficult for ChatGPT to communicate with them in a unified way.

1. Data collection: Data collection is the foundation of an AI system, and the quality of data directly affects subsequent analysis results. However, in practical applications, data collection often faces problems such as missing data, noisy data, and inconsistent data. To ensure the quality and accuracy of collected data, appropriate methods for cleaning and preprocessing must be used.
2. Device control: Device control refers to controlling external devices through an AI system. However, in practical applications, device control often faces problems such as inconsistent device interfaces or communication protocols. Solving these problems requires mastering various device interfaces and communication protocols while designing appropriate control strategies.

That's where Shifu comes in, a new infrastructure that enables LLMs to seamlessly connect with and control IoT devices.

Shifu: The New IoT Infrastructure of LLM Era

Shifu is an IoT development framework that effortlessly connects all of your devices and exposes their APIs, allowing ChatGPT to fully leverage their capabilities. With Shifu, ChatGPT can now complete the loop of "issuing commands to robots or other IoT devices using natural language and having them perform corresponding actions" - bringing large models and the physical world together.

By serving as a bottom-level infrastructure, Shifu enables LLMs to communicate with devices in different ways. With Shifu, LLMs no longer have to worry about the complexity of connecting different devices. Shifu provides LLMs with a unified device API, allowing them to communicate with different devices using natural language. This is achieved by abstracting the different protocols and rules used by different devices. Shifu acts as middleware, translating natural language commands into device-specific commands and vice versa. With Shifu, developers can easily add new devices to their applications without worrying about the underlying complexity.

Shifu is the ultimate solution for extending ChatGPT's capabilities into the physical world. With its unified API, LLMs can now use simple language commands to manipulate heterogeneous devices, freeing them from the hassle of dealing with various protocols and rules. Adding new devices to applications is now a breeze, as Shifu abstracts the complexity of different device workings. In fact, Shifu's integration enables ChatGPT to expand its influence into the physical world, completely changing the way people interact with things by introducing natural language as a new user interface.

Shifu + ChatGPT+IoT = AIoT

Shifu, an open-source AIoT development framework, provides users with a one-stop, complete solution for device access and control. Although Shifu has reduced the difficulty of accessing and controlling devices to the lowest level in the world, users still need to provide communication protocols/drivers and configuration files. Once the types of devices in IoT scenarios become diverse and complex, users may still feel a lot of trouble.

Fortunately, Shifu not only provides convenience for humans, but the unified and standard API it provides can also be easily absorbed and used by ChatGPT. In other words, with ChatGPT and Shifu, device access and use in IoT can be completely transformed into prompt engineering rather than traditional programming. Based on this, we can say that humans have achieved no-code building of AIoT scenes for the first time.

Here is a simple example of using ChatGPT's prompt and Shifu together to complete the connection, drive, and guide a robotic arm to perform a task, starting from scratch:

1. All we need to prepare before starting are the hardware (robotic arm) and a computer. Our goal is to let ChatGPT and Shifu automatically handle the software part. This robotic arm is capable of simple movements and grasping.
2. Firstly, we need to let ChatGPT learn from Shifu's open-source project:

I have a iot frame work called shifu, you can use it to expose the api of iot device. The open-source project is at https://github.com/edgenesis/shifu

After this step, ChatGPT will understand how to generate the DeviceShifu used by Shifu for any hardware device, which is a digital twin microservice. At the same time, it will also learn how to complete a YAML configuration file that describes the user-provided hardware devices.

3. Next, describe the information of the robotic arm to ChatGPT. In this step, ChatGPT will be able to understand how to generate device drivers.

• We need to provide "metadata" about the device, such as its capabilities and connection address:

I have a robotic arm with a suction pump that can pick up objects. You can open the pump and pick up an object by accessing localhost:8050/pump_on, and close the pump and release the object by accessing localhost:8050/pump_off.

The robotic arm itself can also move, and you can send the post request to localhost:8050/send_xyz with the json parameter like {"xyz":[-57.6,-186.8,197.1]}. The array first item is x-axis value, second item is y-axis value, third is z-axis value.

• Next, we need to provide the capability of the device so that ChatGPT can later link API and device capabilities:

Specifically, your python script should include several circle

 

- Move the robotic arm above the block
- Lower the robotic arm to the block and turn on the suction pump
- Lift the robotic arm to 197
- Move the robotic arm to the desired position
- turn off the pump to release the block
- Move the robotic above the block to start new circle

 

**when you move the arm above the block which means you should set the z-axis value with 197 or higher**

 

By these operations, you can move one block into another places, now I will introduce the require for you. you should design a python script for me, which can be run in raspberry. Are you ready?

• We also need to provide our requirements for operating the device and request ChatGPT to write a Python driver for controlling the robotic arm. We can describe from the user's perspective how someone can use this device through the driver provided by ChatGPT. Here, we have given some examples to help ChatGPT better understand our needs:

Requirement

 

1. There are five blocks which are "SHIFU", these five letters are arranged randomly. I will now tell you their coordinates, and you need to rearrange them in order.
2. The blocks have their own position and I will give it for you with an object
   `{s:{-109.5,-157.2,90},h:{-55.7,-167.2,90},u:{-32.3,-235.2,90},i:{-112.4,-219.7,90},f:{-117,-115.2,90}}`
   for one attribute like `{-109.5,-157.2,90}` is x = -109.5 y = -157 z= 90
3. you can adjust the z-axis value to hold up the block.
4. you should move this block into  another side with y-axis > 0
5. your task is to hold up this random arranged block and rearrange them with one line in table (you can implement it with same y-axis value).

 

Now I will give you one example that hold up one block from {-11.4,-160.7,90.1} to {100.8,-170.2,197.1} with shell command. you should design a python program to implement above requirement

curl -X POST -H "Content-Type: application/json" -d '{"xyz":[-11.4,-160.7,197.1]}' localhost:8051/send_xyz
sleep 2
curl -X POST -H "Content-Type: application/json" -d '{"xyz":[-11.4,-160.7,90.1]}' localhost:8051/send_xyz
sleep 1.5
curl localhost:8051/pump_on
sleep 1.5
curl -X POST -H "Content-Type: application/json" -d '{"xyz":[-11.4,-160.7,197.1]}' localhost:8051/send_xyz
sleep 1.5
curl -X POST -H "Content-Type: application/json" -d '{"xyz":[100.8,-170.2,197.1]}' localhost:8051/send_xyz
sleep 1.5
curl -X POST -H "Content-Type: application/json" -d '{"xyz":[100.8,-170.2,95.3]}' localhost:8051/send_xyz
sleep 1.5
curl localhost:8051/pump_off
sleep 2
curl -X POST -H "Content-Type: application/json" -d '{"xyz":[100.8,-170.2,197.1]}' localhost:8051/send_xyz

• The driver is completed. Next, we can start using this driver to generate deviceShifu with ChatGPT.

4. Generating DeviceShifu with ChatGPT is very simple because the open source code of Shifu has provided many rich and complete examples for ChatGPT. We just need to tell it what we want to do.

Now you have the driver for the robot arm. Please write the deviceShifu for this robot arm.

5. Our final step is to have ChatGPT write a YAML configuration file for this robotic arm. Since ChatGPT has already learned from a large number of YAML configuration file samples through Shifu's open source project, we only need to provide the requirements. At the same time, we can have it directly use the metadata we used in the driver before:

Write the config file for this robot arm in the required YAML file format. Re-use our configuration in the previos Python script.

6. Now an AIoT robotic arm operation plan has been completed. Let's take a look at the results it achieved:

<video class="videoBox" autoplay="" loop="" controls="" src="ChatIoT.mp4"></video>