RobotX- Control GUI

Intro

• See the 2nd iteration: RobotX GUI v2

Control GUI v1

The set up for the first iteration of the RobotX control GUI is described.

The web GUI interfaces with ROS2 backend using rosbridge_server over WebSockets for continuous tasks such as Scan the Code light sequence monitoring. Mouse-clicked waypoints on the map are sent as POST HTTP requests to a flask server, which publishes them to ROS. The virtual joystick’s movements are published to \cmd_vel topic as Twist messages for manual control of the robot.

Resources

• https://medium.com/@lamthaithanhlong/building-a-remote-controlled-robot-with-ros-flask-and-a-custom-web-interface-78db786572f6
• https://wiki.ros.org/vizanti
• Campus rover- Integrating using Flask and ROS
• YT- A web application for Navigation and Mapping ROS | app.py

greenhorn_gui package

Package Creation and Structure

The greenhorn_gui package was created with the following steps:

• Create/checkout rosbridge_gui branch

cd mrg/robotx_ws/src/greenhorn
git checkout -b rosbridge_gui

• Generate the greenhorn_gui package:

ros2 pkg create greenhorn_gui 

• Add additional files in the following structure:

greenhorn_gui structure

• Hover over filename to preview:
  greenhorn_gui/
  ├── CMakeLists.txt
  ├── package.xml
  ├── gui/
  │ ├── index.html
  │ ├── index.js
  │ └── style.css
  ├── launch/
  │ ├── light_buoy_sim_launch.py
  │ └── websocket.launch.py
  └── light_buoy_sim/
  │ └── light_buoy_publisher.py
  └── server/
  └── app.py

• CMakeLists.txt: Handles the build system for the package.
• package.xml: Defines package metadata and dependencies.
• gui/: Contains the front-end files (HTML, JavaScript, CSS) that make up the user interface.
• launch/: Includes launch files to start the simulation and set up the WebSocket connection.
• light_buoy_sim/: The ROS2 node for simulating light buoy data.
• server/: Contains the backend logic (app.py), which acts as both a Flask API and a ROS2 publisher.

CMakeLists.txt

CMakeLists.txt

cmake_minimum_required(VERSION 3.8)
project(greenhorn_gui)
 
if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
  add_compile_options(-Wall -Wextra -Wpedantic)
endif()
 
# find dependencies
find_package(ament_cmake REQUIRED)
find_package(rosbridge_server REQUIRED)
 
if(BUILD_TESTING)
  find_package(ament_lint_auto REQUIRED)
  # the following line skips the linter which checks for copyrights
  # comment the line when a copyright and license is added to all source files
  set(ament_cmake_copyright_FOUND TRUE)
  # the following line skips cpplint (only works in a git repo)
  # comment the line when this package is in a git repo and when
  # a copyright and license is added to all source files
  set(ament_cmake_cpplint_FOUND TRUE)
  ament_lint_auto_find_test_dependencies()
endif()
 
ament_package()
 
# Install Python scripts
install(PROGRAMS
  light_buoy_sim/light_buoy_publisher.py
  DESTINATION lib/${PROJECT_NAME}
)
 
# Install server folder
install(DIRECTORY server/
DESTINATION share/${PROJECT_NAME}/server)
 
# Install other launch files
install(
  DIRECTORY launch/
  DESTINATION share/${PROJECT_NAME}/
)

package.xml

package.xml: Defines package metadata and dependencies.

package.xml

<?xml version="1.0"?>  
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>  
 
<package format="3">  
  <name>greenhorn_gui</name>  
  <version>0.0.0</version>  
  <description>Greenhorn GUI</description>  
  <license>TODO: License declaration</license>  
 
  <!-- Build tool dependencies -->  
  <buildtool_depend>ament_cmake</buildtool_depend>  
 
  <!-- Execution dependencies -->  
  <exec_depend>rclpy</exec_depend>  
  <exec_depend>std_msgs</exec_depend>  
  <exec_depend>python3-flask</exec_depend>  
  
  <!-- General dependencies -->  
  <depend>rosbridge_server</depend>  
 
  <!-- Testing dependencies -->  
  <test_depend>ament_lint_auto</test_depend>  
  <test_depend>ament_lint_common</test_depend>  
 
  <!-- Export section -->  
  <export>  
    <build_type>ament_cmake</build_type>  
  </export>  
</package>  

Build the package

Ensure all necessary packages are present before building the greenhorn_gui package

• Install all required dependencies from the root of the workspace

cd ../..
sudo apt update
rosdep install --from-paths src --ignore-src -r -y

• Build the workspace

colcon build --packages-select greenhorn_gui

---

Rosbridge Server

websocket.launch.py

websocket.launch.py

from launch import LaunchDescription
from launch_ros.actions import Node
from launch.actions import ExecuteProcess
from ament_index_python.packages import get_package_share_directory
 
def generate_launch_description():
    package_dir = get_package_share_directory('greenhorn_gui')
    return LaunchDescription([
        Node(
            package='rosbridge_server',
            executable='rosbridge_websocket',
            name='rosbridge_websocket',
            output='screen',
            parameters=[{'port': 9090}]
        ),
        ExecuteProcess(
            cmd=['python3', f'{package_dir}/server/app.py'],
            output='screen'
        )
    ])
`

Note: this file should be renamed as it launches both the rosbridge server node and the flask node

• Launch the websocket and flask server

source install/setup.bash
ros2 launch greenhorn_gui websocket_launch.py

[INFO] [launch]: All log files can be found below /home/chxtio/.ros/log/2024-09-24-07-01-23-452406-172.20225150-1507
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [rosbridge_websocket-1]: process started with pid [1509]
[rosbridge_websocket-1] [INFO] [1727186484.038599832] [rosbridge_websocket]: Rosbridge WebSocket server started on port 9090

---

Web GUI

gui/: Contains the front-end files (HTML, JavaScript, CSS) that make up the user interface.

index.html

• Global map added using maplibre

index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>RobotX GUI</title>
    <link rel="stylesheet" href="style.css">
    <link href="https://fonts.googleapis.com/css2?family=Roboto:wght@700&family=Montserrat:wght@700&display=swap" rel="stylesheet">
    <script src="https://cdn.jsdelivr.net/npm/roslib@1/build/roslib.min.js"></script>
    <link href="https://unpkg.com/maplibre-gl@2.1.9/dist/maplibre-gl.css" rel="stylesheet" />
    <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/nipplejs/0.7.3/nipplejs.js"></script>
  </head>
 
  <body>
    <div id="map_label">
      <a id="map_link" href="" target="_blank">
         📌 
      </a>
      <h1 id="map_coords_label">
      </h1>
    </div>
 
    <!-- <div id="map" style="width: 100%; height: 500px;"></div> -->
    <div id="map" style="width: 450px; height: 350px; margin: auto;"></div>
 
    <script src="https://unpkg.com/maplibre-gl@2.1.9/dist/maplibre-gl.js"></script>
 
    <div id="test-box">
      <div id="img-div">
      </div>
      <div id="buoy_div" class="hidden"></div>
      <p id="buoy_label" class="hidden">Light Tower</p>
      <!-- <button onclick="sendCommand()">Send Command</button> -->
      <!-- Sample button that can be used for another command -->
      <button id="toggleBuoy">Show</button>
    </div>s
 
    <canvas id="canvas"></canvas>
    <div class="center">
        <div class="centered-text">
            <h1 class="inline-text">Scan the Code </h1> 
            <span id="status" class="inline-text"></span>
        </div>
 
      <div class="grid-container">
        <div class="scan_code_label" id="label1"></div>
        <div class="scan_code_label" id="label2"></div>
        <div class="scan_code_label" id="label3"></div>
        <div class="box-black" id="box1"></div>
        <div class="box-black" id="box2"></div>
        <div class="box-black" id="box3"></div>
      </div>
      <sim-controls></sim-controls>
    </div>
 
    <div id="zone_joystick"></div>
 
    <script src="index.js"></script>
 
  </body>
</html>

style.css

style.css

body {
    background-color: #00274d; 
    color: #f0f0f0;
    font-family: 'Poppins', sans-serif;
}
h1 {
    text-align: center;
    margin-bottom: 15px;
}
h1:first-of-type {
    font-size: 2.5rem;
    font-weight: 800;
    color: #ffd700; 
    text-shadow: 2px 2px 5px rgba(0, 0, 0, 0.4);
    letter-spacing: 3px;
}
h1:last-of-type {
    font-size: 1.5rem;
    font-weight: 700;
    color: #ffd700; 
    text-shadow: 1px 1px 3px rgba(0, 0, 0, 0.5);
    letter-spacing: 1px;
}
 
.center {
    margin: auto;
    width: fit-content;
    padding: 10px;
}
 
/* .box {
    text-align: center;
    width: fit-content;
    margin-left: 10px;
    margin-top: 10px;
    padding: 5px;
    background-color: white;
    border-radius: 15px;
} */
 
.btn-group button {
    background-color: grey;
    border: 1px solid grey;
    color: white;
    padding: 10px 24px;
    cursor: pointer;
    float: center;
    border-radius: 10px;
}
 
.btn-group:after {
    content: "";
    clear: both;
    display: table;
}
 
.btn-group button:not(:last-child) {
    border-right: none;
}
 
.btn-group button:hover {
    background-color: lightgray;
}
 
#canvas {
    margin-top: -150px; 
    display: block; 
    margin-left: auto;
    margin-right: auto; 
}
 
.grid-container {
    display: grid;
    grid-template-columns: repeat(3, 1fr); 
    grid-gap: 5%; /* add a 10px gap between the boxes */
    margin: auto;
    margin-right: 100px;
    width: fit-content;
    /* padding: 10px; */
}
 
.grid-item {
    /* background-color: #ccc; */
    padding: 20px;
    text-align: center;
}
 
#status {
    padding: 5px;
    border-radius: 5px;
    color: white;
    font-weight: bold;
}
 
#sample_msg {
    padding: 5px;
    border-radius: 5px;
    font-weight: bold;
}
 
#buoy_div {
    padding: 5px;
    border: 10px solid white;
    /* border-color: white; */
    border-radius: 0px;
    font-weight: bold;
    width: 20px;
    height: 30px;
    margin-left: 40px;
}
 
#toggleBuoy {
    margin-left: 40px;
    margin-top: 10px;
}
 
.inline-text {
    display: inline-block;
    margin-right: 10px; 
    vertical-align: middle; 
}
 
.centered-text {
    display: flex;
    justify-content: center;
    align-items: center;
}
 
/* Colors for different statuses */
.status-connected {
    background-color: green;
}
 
.status-disconnected {
    background-color: red;
}
 
.status-error {
    background-color: orange;
}
 
/* Color for message label */
.message-received {
    background-color: lightblue;
    color: black;
}
 
/* Set the color of the boxes */
[class^="box-"] {
    text-align: center;
    width: 3cm;
    height: 3cm;
    /* width: 200px;
    height: 200px; */
    margin-top: 10px;
    padding: 5px;
    border: 2px solid black;
}
 
.box-red {
    background-color: red;
}
 
.box-blue {
    background-color: blue;
}
 
.box-green {
    background-color: green;
}
 
.box-black {
    background-color: black;
}
 
.scan_code_label {
    text-align: center;
    margin: auto;
    margin-top: 10px;
    /* width: 200px;
    height: 30px;
    line-height: 30px; */
    width: 3cm;
    height: 1cm;
    line-height: 1cm;
    padding: 5px;
    font-size: 20px; 
    color: black;
    background-color: white;
    border-radius: 0px;
    border: 2px solid black;
}
 
ul {
    margin: 0;
    padding: 0;
    list-style: none;
}
 
.deploy-status {
    &.open:before {
        background-color: #94E185;
        border-color: #78D965;
        box-shadow: 0px 0px 4px 1px #94E185;
    }
 
    &.in-progress:before {
        background-color: #FFC182;
        border-color: #FFB161;
        box-shadow: 0px 0px 4px 1px #FFC182;
    }
    
    &.dead:before {
        background-color: #C9404D;
        border-color: #C42C3B;
        box-shadow: 0px 0px 4px 1px #C9404D;
    }
 
    &:before {
        content: ' ';
        display: inline-block;
        width: 7px;
        height: 7px;
        margin-right: 10px;
        border: 1px solid #000;
        border-radius: 7px;
    }
}
 
.hidden {
    display: none;
}
 
#test-box {
    color: gold;
    width: 250px;
    text-align: left;
    padding: 5px;
    margin-top: 20px; 
    /* display: none; */
}
 
#img-div {
    display: flex;
    justify-content: center;
}
 
/* Map */
#map_label {
    display: flex;
    align-items: center;
    justify-content: center;
}
 
#zone_joystick {
    position: absolute;
    /* bottom: 20px;   */
    top: 300px;
    right: 150px;   
    width: 100px;  
    height: 100px; 
    z-index: 10;   
}

index.js

index.js

// var ws = new WebSocket('ws://localhost:9090');
// ws.onopen = function() {
//     console.log('WebSocket connection opened.');
// };
// ws.onerror = function(error) {
//     console.error('WebSocket error: ', error);
// };
// ws.onmessage = function(event) {
//     console.log('WebSocket message: ', event.data);
// }; 
 
const statusLabel = document.getElementById('status');
const buoy_div = document.getElementById('buoy_div');
const buoy_label = document.getElementById('buoy_label');
const toggle_button = document.getElementById('toggleBuoy');
 
const light1 = document.getElementById('box1');
const label1 = document.getElementById('label1');
const light2 = document.getElementById('box2');
const label2 = document.getElementById('label2');
const light3 = document.getElementById('box3');
const label3 = document.getElementById('label3');
 
const map_div = document.getElementById('map');
const map_label = document.getElementById('map_label');
const map_coords_label = document.getElementById('map_coords_label');
 
var ros = new ROSLIB.Ros({
    url: 'ws://localhost:9090'
});
 
 
ros.on('connection', function() {
    statusLabel.className = "deploy-status open";
    console.log('Connected to websocket server.');
});
 
ros.on('error', function(error){
    document.getElementById("status").innerHTML = "Error connecting to ROS";
    statusLabel.className = 'status-error';
    console.log('Error connecting to websocket server.');
})
 
ros.on('close', function() {
    statusLabel.className = 'deploy-status dead';
    console.log('Closed connection to websocket server.');
});
 
// Subscribe to a light tower topic
var listener = new ROSLIB.Topic({
    ros: ros,
    name: "/light_sequence",
    messageType: "std_msgs/String",
});
 
var box_colors = ['black', 'black', 'black'];
 
listener.subscribe(function (message){
    var incomingLight = message.data;
    // console.log("Received message: " + incomingLight);
    var box_color = "box-" + incomingLight;
    if (!buoy_div.classList.contains('hidden') ){
        buoy_div.classList = box_color;
    }
 
 
    if (incomingLight !== "black") {
        if (box_colors[0] === "black") {
            box_colors[0] = incomingLight;
        } else if (box_colors[1] === "black") {
            box_colors [1] = incomingLight;
        } else if (box_colors[2] === "black") {
            box_colors[2] = incomingLight;
        }
    } else {
        // Reset all boxes to black after the sequence
        box_colors = ['black', 'black', 'black']
    }
 
    var labels = [label1, label2, label3];
    var lights = [light1, light2, light3];
 
    for (let i = 0; i < 3; i++) {
        var color = box_colors[i];
        if (color !== "black") {
            labels[i].textContent = color.toUpperCase();
        } else {
            labels[i].textContent = " ";
        }
        lights[i].className = "box-" + color;
    }
});
 
 
function toggleBuoy(){
    // console.log(buoy_div.textContent); 
    if (buoy_div.classList.contains('hidden')){
        buoy_div.classList.remove('hidden');
        // buoy_label.classList.remove('hidden');
        toggle_button.textContent = "Hide";
    } else {
        buoy_div.classList.add('hidden');
        // buoy_label.classList.add('hidden');
        toggle_button.textContent = "Show";
    }
}
 
toggle_button.addEventListener('click', toggleBuoy);
 
// Map functions
// 0) demo 1) OSM-based map style
const map_styles = ['https://demotiles.maplibre.org/style.json', 'https://tiles.stadiamaps.com/styles/alidade_smooth.json']
var placeName = "Nathan Benderson Park, FL"
var map_url = "https://www.google.com/maps/place/Nathan+Benderson+Park/@27.3742443,-82.4500873,15z/data=!4m6!3m5!1s0x88c338bd14033973:0xba4efc2b7130fac1!8m2!3d27.3742443!4d-82.4500873!16s%2Fg%2F11f11nr15f?entry=ttu&g_ep=EgoyMDI0MDkyNC4wIKXMDSoASAFQAw%3D%3D";
var coordinates = [-82.4500873, 27.3742443];
 
 
const map = new maplibregl.Map({
    container: 'map',
    style: map_styles[1],
    // center: [0, 0],  // Longitude, Latitude
    center: coordinates,
    zoom: 14
  });
 
map.on('click', (event) => {
    const lngLat = event.lngLat;
    console.log('New waypoint:', lngLat);
 
    // Add waypoint
    new maplibregl.Marker()
    .setLngLat(lngLat)
    .addTo(map);
 
    // Send waypoint to Flask server
   const waypoint = {
    lng: lngLat.lng,  
    lat: lngLat.lat   
    };
 
    fetch('http://localhost:5000/waypoints', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify(waypoint)  
    })
    .then(response => {
        if (!response.ok) {
            throw new Error(`HTTP error! status: ${response.status}`);
        }
        return response.json();
    })
    .then(data => console.log('Success:', data))
    .catch(error => console.error('Error:', error));
});
 
// Update map labels
map_coords_label.textContent = placeName + " (" + `${coordinates[0]}, ${coordinates[1]}` + ")";
document.getElementById('map_link').href = map_url;
 
// Publish command velocity
cmd_vel_listener = new ROSLIB.Topic({
    ros : ros,
    name : "/cmd_vel",
    messageType : 'geometry_msgs/Twist'
  });
   
  move = function (linear, angular) {
    var twist = new ROSLIB.Message({
      linear: {
        x: linear,
        y: 0,
        z: 0
      },
      angular: {
        x: 0,
        y: 0,
        z: angular
      }
    });
    cmd_vel_listener.publish(twist);
  }
 
// Create virutal joystick object and handle events
createJoystick = function () {
    var options = {
      zone: document.getElementById('zone_joystick'),
      threshold: 0.1,
      position: { left: '50%' },
      mode: 'static',
      size: 150,
      color: '#000000',
    };
    manager = nipplejs.create(options);
   
    linear_speed = 0;
    angular_speed = 0;
 
    manager.on('start', function (event, nipple) {
        timer = setInterval(function () {
          move(linear_speed, angular_speed);
        }, 25);
      });
   
    manager.on('end', function () {
        if (timer) {
          clearInterval(timer);
        }
        move(0, 0); // Stop the robot
      });
   
      manager.on('move', function (event, nipple) {
        max_linear = 5.0; // Max linear velocity (m/s)
        max_angular = 2.0; // Max angular velocity (rad/s)
        max_distance = 75.0; // Max joystick distance (pixels)
       
        linear_speed = Math.sin(nipple.angle.radian) * max_linear * nipple.distance / max_distance;
        angular_speed = -Math.cos(nipple.angle.radian) * max_angular * nipple.distance / max_distance;
      });
   
    manager.on('end', function () {
      console.log("Movement end");
    });
  }
   
  window.onload = function () {
    createJoystick();
  }

---

Light Tower Simulator

light_buoy_sim_launch.py

light_buoy_sim_launch.py

from launch import LaunchDescription
from launch_ros.actions import Node
 
def generate_launch_description():
    return LaunchDescription([
        Node(
            package='greenhorn_gui',
            executable='light_buoy_publisher.py',
            name='light_buoy_publisher',
            output='screen'
        )
    ])

• Launch the light buoy simulator

source install/setup.bash
ros2 launch greenhorn_gui light_buoy_sim_launch.py

---

Flask REST API with ROS Integration

server/: Contains app.py, which acts as both a Flask API for http endpoints and a ROS2 publisher

app.py

app.py

from flask import Flask, request, jsonify#, render_template_string
from flask_cors import CORS
import rclpy
from rclpy.node import Node
from std_msgs.msg import Float64MultiArray
 
app = Flask(__name__)
CORS(app) # Allow communication with different ports
 
waypoints = []
 
class WaypointPublisher(Node):
    def __init__(self):
        super().__init__('waypoint_publisher')
        self.publisher_ = self.create_publisher(Float64MultiArray, '/waypoints', 10)
 
    def publish_waypoint(self, waypoint):
        msg = Float64MultiArray()
        msg.data = [waypoint['lng'], waypoint['lat']]
        self.publisher_.publish(msg)
        self.get_logger().info(f'Published waypoint: {msg.data}')
 
# Initialize ROS node
rclpy.init()
waypoint_publisher = WaypointPublisher()
 
@app.route("/")
def index():
    return "<p>Hello, add waypoints to the map via the API</p>"
 
@app.route('/waypoints', methods=['POST'])
def process_waypoints():
    data = request.json
    waypoints.append(data)
 
    # Publish waypoint to ROS topic
    waypoint_publisher.publish_waypoint(data)
 
    return jsonify({'status': 'success', 'message': 'Waypoint received', 'waypoint': data}), 200
 
if __name__ == "__main__":
    try:
        app.run(host='0.0.0.0', port=5000)
    except KeyboardInterrupt:
        pass
    finally:
        waypoint_publisher.destroy_node()
        rclpy.shutdown()

Note

The Flask node is launched via websocket.launch.py

• To see a demonstration (table of waypoints) of how the GUI can be served using Python with Flask and templating (as opposed to using javascript and rosbridge), visit the URL:

http://127.0.0.1:5000/

Publish waypoints to ROS

• Test publisher by either clicking points on the map or sending a POST request to /waypoints topic via the terminal:

curl -X POST http://localhost:5000/waypoints -H "Content-Type: application/json" -d '{"lng": 1.0, "lat": 2.0}'

• Subscribe to /waypoints topic to listen to the messages

ros2 topic echo /waypoints

 ros2 topic echo /waypoints
layout:
  dim: []
  data_offset: 0
data:
- -82.44391607402257
- 27.37299715058211
---
layout:
  dim: []
  data_offset: 0
data:
- -82.45082544444473
- 27.376884323360812
---
layout:
  dim: []
  data_offset: 0
data:
- -82.449237576709
- 27.370100736993308

---

Virtual Joystick

• See virtual-joystick for more info

• The move function takes in 2 arguments: linear and angular speed in m/s and publishes it as Twist messages on cmd_vel topic through the cmd_vel_listener object

// Publish command velocity
cmd_vel_listener = new ROSLIB.Topic({
    ros : ros,
    name : "/cmd_vel",
    messageType : 'geometry_msgs/Twist'
  });
   
  move = function (linear, angular) {
    var twist = new ROSLIB.Message({
      linear: {
        x: linear,  // forward/backward
        y: 0,       // side-to-side 
        z: 0        // up/down 
      },
      angular: {
        x: 0,       // roll (rotation around x-axis)
        y: 0,       // pitch (rotation around y-axis)
        z: angular  // yaw (rotation around z-axis)
      }
    });
    cmd_vel_listener.publish(twist);
  }

Note: Name should be changed to reflect the Topic object as a publisher

---

GUI v1 fixes

Load dependencies locally for offline use

• Download and serve dependencies locally from a /libs folder for offline access

  • roslibjs- pre-built files no longer included

    • Using NPM: npm install roslib
    • CDN: https://cdn.jsdelivr.net/npm/roslib@1/build/roslib.min.js

  • maplibre

    • Download files from repo:
      • maplibre-gl.js
      • maplibre-gl.css

  • Virtual Joystick

    • CDN: https://cdnjs.cloudflare.com/ajax/libs/nipplejs/0.7.3/nipplejs.js

• Update index.html

    <script type="text/javascript" src="libs/roslib.min.js"></script>
    <link href="libs/maplibre-gl.css" rel="stylesheet" />
    <script src="libs/maplibre-gl.js"></script>
    <script type="text/javascript" src="libs/nipplejs.js"></script>