• Install Python: Make sure Python is set up on your computer. Or you can use google colab for an amazing experience and create a Python script.

• Install Required Libraries: Open your command line interface and install Gym and PyTorch by running:

pip install gym torch

• Set Up the Environment. We'll use an essential Gym environment for our example. This script creates an environment where an agent moves in a one-dimensional space.

import gym
import numpy as np


env = gym.make('MountainCarContinuous-v0',  new_step_api=True)


# This function represents a step taken by the agent
def step(action):
    observation, reward, done, truncated, info = env.step(action)
    return observation, reward, done, truncated

• Now, define a Basic Agent. Here, we'll define a simple agent that makes random moves. In a more advanced setup, this would be replaced with a neural network.

def basic_agent(observation):
    return [np.random.uniform(-1, 1)]

• Implement Human Feedback. For simplicity, our "human feedback" will be a function that adjusts the reward based on the agent's position.

def human_feedback(observation, reward):
    # Increase reward if the agent is closer to the target position
    target_position = 0.45
    if observation[0] > target_position:
        reward += 10
    return reward

• Run the Training Loop. Now, let's put it all together in a training loop. Usually, this would involve learning from the environment, but our agent will just make random moves for simplicity.

max_timesteps_per_episode = 200  # Set a limit to the number of timesteps per episode


for episode in range(10):
    observation = env.reset()
    total_reward = 0


    for t in range(max_timesteps_per_episode):
        action = basic_agent(observation)
        observation, reward, done, truncated = step(action)
        reward = human_feedback(observation, reward)
        total_reward += reward


        if done or truncated:
         print(f"Episode {episode + 1} finished after {t + 1} timesteps, total reward: {total_reward}")
            break
        if t == max_timesteps_per_episode - 1:
           print(f"Episode {episode + 1} reached the maximum timesteps, total reward: {total_reward}")

• Run this script in your Python environment. You should see output for each episode indicating how the agent performed. It will give you output like this one:

There is no need to worry about these negative values. In the MountainCarContinuous-v0 environment from OpenAI Gym, the reward structure can result in negative values, especially with how the environment and tasks are designed. This environment is unique compared to others because of its reward system. Here are a few key points about its reward structure:

Goal of the Environment: The primary objective in the MountainCarContinuous-v0 environment is to reach the top of a hill. However, the car's engine isn't strong enough to climb the mountain in a straight line, so it must build momentum by going back and forth.

Default Reward System: The reward in this environment is usually structured to encourage reaching the goal with minimal action. A typical reward structure is to give a small negative bonus (penalty) for each time step until the goal is reached. This means the longer it takes to reach the plan, the more negative the total reward will be.

Impact of Human Feedback: In the context of RLHF, you can adjust this reward system. For instance, you might give additional positive rewards (or reduce penalties) when the car moves in the right direction or achieves intermediate goals, like reaching a certain height on the hill.

Possibility of Negative Rewards: Due to the nature of the task and the default reward system, it's common to see negative rewards in this environment. The key is to reduce the number of steps (and thus negative tips) needed to reach the goal or to adjust the reward system to guide the learning process better.