torchdrivesim.simulator ======================= .. py:module:: torchdrivesim.simulator Attributes ---------- .. autoapisummary:: torchdrivesim.simulator.logger torchdrivesim.simulator.TensorPerAgentType torchdrivesim.simulator.IntPerAgentType Classes ------- .. autoapisummary:: torchdrivesim.simulator.CollisionMetric torchdrivesim.simulator.TorchDriveConfig torchdrivesim.simulator.AgentTypeFunctor torchdrivesim.simulator.SingletonAgentTypeFunctor torchdrivesim.simulator.DictAgentTypeFunctor torchdrivesim.simulator.SimulatorInterface torchdrivesim.simulator.Simulator torchdrivesim.simulator.SimulatorWrapper torchdrivesim.simulator.NPCWrapper torchdrivesim.simulator.HomogeneousWrapper torchdrivesim.simulator.RecordingWrapper torchdrivesim.simulator.BirdviewRecordingWrapper torchdrivesim.simulator.TrajectoryVisualizationWrapper torchdrivesim.simulator.SelectiveWrapper torchdrivesim.simulator.BoundedRegionWrapper torchdrivesim.simulator.NoReentryBoundedRegionWrapper Module Contents --------------- .. py:data:: logger .. py:class:: CollisionMetric Bases: :py:obj:`enum.Enum` Method used to calculate collisions between agents. .. py:attribute:: iou :value: 'iou' .. py:attribute:: discs :value: 'discs' .. py:attribute:: nograd :value: 'nograd' .. py:attribute:: nograd_pytorch3d :value: 'nograd-pytorch3d' .. py:class:: TorchDriveConfig Top-level configuration for a TorchDriveSim simulator. .. py:attribute:: renderer :type: torchdrivesim.rendering.RendererConfig .. py:attribute:: single_agent_rendering :type: bool :value: False .. py:attribute:: collision_metric :type: CollisionMetric .. py:attribute:: offroad_threshold :type: float :value: 0.5 .. py:attribute:: left_handed_coordinates :type: bool :value: False .. py:attribute:: wrong_way_angle_threshold :type: float .. py:attribute:: lanelet_inclusion_tolerance :type: float :value: 1.0 .. py:attribute:: waypoint_removal_threshold :type: float :value: 2.0 .. py:data:: TensorPerAgentType .. py:data:: IntPerAgentType .. py:class:: AgentTypeFunctor Bases: :py:obj:`abc.ABC` Lifts functions operating on individual agent type to functions operating across all agent types. Children of this class specify how the collection of agents is represented. .. py:method:: fmap(f, *args) :abstractmethod: Applies a given function of any number of arguments to all agent types. .. py:method:: to_device(tensor: TensorPerAgentType, device: torch.device) -> TensorPerAgentType Applies `.to` to each element. .. py:method:: __call__(f, *args) .. py:class:: SingletonAgentTypeFunctor Bases: :py:obj:`AgentTypeFunctor` Trivial functor where only one agent type is used. No packaging is used for the arguments. .. py:method:: fmap(f, *args) Applies a given function of any number of arguments to all agent types. .. py:class:: DictAgentTypeFunctor(agent_types: List[str]) Bases: :py:obj:`AgentTypeFunctor` Arguments are packaged as dictionaries, with keys being agent type names. .. py:method:: fmap(f, *args) Applies a given function of any number of arguments to all agent types. .. py:class:: SimulatorInterface Abstract interface for a 2D differentiable driving simulator. .. py:property:: agent_functor :type: AgentTypeFunctor :abstractmethod: Defines how to apply functions across agent types. .. py:property:: agent_types :type: Optional[List[str]] :abstractmethod: List of agent types used by this simulator, or `None` if only one type used. .. py:method:: across_agent_types(f, *args) Applies a given per-agent-type operation across all agent types. Subsequent arguments should be given in the format of multiple agent collection that this class uses, which is typically a dictionary with keys being agent type names. .. py:property:: action_size :type: IntPerAgentType :abstractmethod: Defines the size of the action space for each agent type. .. py:property:: batch_size :type: int :abstractmethod: .. py:method:: to(device) -> typing_extensions.Self :abstractmethod: Modifies the simulator in-place, putting all tensors on the device provided. .. py:method:: copy() -> typing_extensions.Self :abstractmethod: Duplicates this simulator, allowing for independent subsequent execution. The copy is relatively shallow, in that the tensors are the same objects but dictionaries referring to them are shallowly copied. .. py:method:: extend(n: int, in_place: bool = True) -> typing_extensions.Self :abstractmethod: Multiplies the first batch dimension by the given number. Like in pytorch3d, this is equivalent to introducing extra batch dimension on the right and then flattening. .. py:method:: select_batch_elements(idx, in_place=True) -> typing_extensions.Self :abstractmethod: Picks selected elements of the batch. The input is a tensor of indices into the batch dimension. .. py:method:: __getitem__(item: torch.Tensor) -> typing_extensions.Self Allows indexing syntax. `item` should be an iterable collection of indices. .. py:property:: agent_count :type: IntPerAgentType How many agents of each type are there in the simulation. This counts the available slots, not taking present masks into consideration. .. py:method:: get_world_center() -> torch.Tensor :abstractmethod: Returns a Bx2 tensor with the coordinates of the map center. .. py:method:: get_state() -> TensorPerAgentType :abstractmethod: Returns a functor of BxAxSt tensors representing current agent states. .. py:method:: get_agent_size() -> TensorPerAgentType :abstractmethod: Returns a functor of BxAx2 tensors representing agent length and width. .. py:method:: get_agent_type() -> TensorPerAgentType :abstractmethod: Returns a functor of BxA long tensors containing agent type indexes relative to the list containing all agent types as returned by `SimulatorInterface.agent_types`. .. py:method:: get_present_mask() -> TensorPerAgentType :abstractmethod: Returns a functor of BxA boolean tensors indicating which agents are currently present in the simulation. .. py:method:: get_all_agents_absolute() -> TensorPerAgentType :abstractmethod: Returns a functor of BxAx6 tensors, where the last dimension contains the following information: x, y, psi, length, width, present. Typically used to implement non-visual observation modalities. .. py:method:: get_all_agents_relative(exclude_self: bool = True) -> TensorPerAgentType :abstractmethod: Returns a functor of BxAxAllx6 tensors, specifying for each of A agents the relative position about the other agents. 'All' is the number of all agents in the simulation, including hidden ones, across all agent types. If `exclude_self` is set, for each agent in A, that agent itself is removed from All. The final dimension has the same meaning as in `get_all_agents_absolute`, except now the positions and orientations are relative to the specified agent. .. py:method:: get_traffic_controls() -> Dict[str, torchdrivesim.traffic_controls.BaseTrafficControl] Produces all traffic controls existing in the simulation, grouped by type. .. py:method:: get_innermost_simulator() -> typing_extensions.Self :abstractmethod: Returns the innermost concrete Simulator object. The type signature is misleading due to Python limitations. .. py:method:: get_waypoints() -> TensorPerAgentType :abstractmethod: Returns a functor of BxAxMx2 tensors representing current agent waypoints. .. py:method:: get_waypoints_state() -> TensorPerAgentType :abstractmethod: Returns a functor of BxAx1 tensors representing current agent waypoints state. .. py:method:: get_waypoints_mask() -> TensorPerAgentType :abstractmethod: Returns a functor of BxAxM boolean tensors representing current agent waypoints present mask. .. py:method:: step(agent_action: TensorPerAgentType) -> None :abstractmethod: Runs the simulation for one step with given agent actions. Input is a functor of BxAxAc tensors, where Ac is determined by the kinematic model. .. py:method:: set_state(agent_state: TensorPerAgentType, mask: Optional[TensorPerAgentType] = None) -> None :abstractmethod: Arbitrarily set the state of the agents, without advancing the simulation. The change is effective immediately, without waiting for the next step. :param agent_state: a functor of BxAx4 tensors with agent states :param mask: a functor of BxA boolean tensors, deciding which agent states to update; all by default .. py:method:: update_present_mask(present_mask: TensorPerAgentType) -> None :abstractmethod: Sets the present mask of agents to the provided value. :param present_mask: a functor of BxA boolean tensors .. py:method:: fit_action(future_state: TensorPerAgentType, current_state: Optional[TensorPerAgentType] = None) -> TensorPerAgentType :abstractmethod: Computes an action that would (aproximately) produce the desired state. :param future_state: a functor of BxAx4 tensors defining the desired state :param current_state: if different from the current simulation state, in the same format as future state :returns: a functor of BxAxAc tensors .. py:method:: render(camera_xy: torch.Tensor, camera_psi: torch.Tensor, res: Optional[torchdrivesim.utils.Resolution] = None, rendering_mask: Optional[TensorPerAgentType] = None, fov: Optional[float] = None, waypoints: Optional[torch.Tensor] = None, waypoints_rendering_mask: Optional[torch.Tensor] = None) -> torch.Tensor :abstractmethod: Renders the world from bird's eye view using cameras in given positions. :param camera_xy: BxNx2 tensor of x-y positions for N cameras :param camera_psi: BxNx1 tensor of orientations for N cameras :param res: desired image resolution (only square resolutions are supported; by default use value from config) :param rendering_mask: functor of BxNxA tensors, indicating which agents should be rendered each camera :param fov: the field of view of the resulting image in meters (by default use value from config) :param waypoints: BxNxMx2 tensor of `M` waypoints per camera (x,y) :param waypoints_rendering_mask: BxNxM tensor of `M` waypoint masks per camera, indicating which waypoints should be rendered :returns: BxNxCxHxW tensor of resulting RGB images for each camera .. py:method:: render_egocentric(ego_rotate: bool = True, res: Optional[torchdrivesim.utils.Resolution] = None, fov: Optional[float] = None) -> TensorPerAgentType Renders the world using cameras placed on each agent. :param ego_rotate: whether to orient the cameras such that the ego agent faces up in the image :param res: desired image resolution (only square resolutions are supported; by default use value from config) :param fov: the field of view of the resulting image in meters (by default use value from config) :returns: a functor of BxAxCxHxW tensors of resulting RGB images for each agent. .. py:method:: compute_offroad() -> TensorPerAgentType :abstractmethod: Offroad metric for each agent, defined as the distance to the road mesh. See `torchdrivesim.infractions.offroad_infraction_loss` for details. :returns: a functor of BxA tensors .. py:method:: compute_wrong_way() -> TensorPerAgentType :abstractmethod: Wrong-way metric for each agent, based on the inner product between the agent and lane direction. See `torchdrivesim.infractions.lanelet_orientation_loss` for details. :returns: a functor of BxA tensors .. py:method:: compute_traffic_lights_violations() -> TensorPerAgentType Boolean value indicating whether each agent is committing a traffic light violation. See `torchdrivesim.infractions.traffic_controls.TrafficLightControl.compute_violations` for details. :returns: a functor of BxA tensors .. py:method:: _compute_collision_of_single_agent(box: torch.Tensor, remove_self_overlap: Optional[torch.Tensor] = None, agent_types: Optional[List[str]] = None) -> torch.Tensor :abstractmethod: Computes the collision metric for an agent specified as a bounding box. Includes collisions with all agents in the simulation, including the ones not exposed through the interface of this class. Used with `discs` and `iou` metrics. :param box: Bx5 tensor, with the last dimension being (x,y,length,width,psi). :param remove_self_overlap: B boolean tensor, where if the input agent is present in the simulation, set this to subtract self-overlap. By default it is assumed that self overlapping exists and will be removed. :param agent_types: An optional list of specific agent types for computing collisions with. By default all available agent types will be used. :returns: a tensor with a single dimension of B elements .. py:method:: _compute_collision_of_multi_agents(mask: Optional[torch.Tensor] = None) -> TensorPerAgentType :abstractmethod: Computes the collision metric for selected (default all) agents in the simulation. Includes collisions with all agents in the simulation, including the ones not exposed through the interface of this class. Used with `nograd` and `nograd-pytorch3d` metrics. :param mask: a functor of BxA boolean tensors, indicating for which agents to compute the loss (by default use present mask) :returns: a functor of BxA tensors .. py:method:: compute_collision(agent_types: Optional[List[str]] = None) -> TensorPerAgentType Compute the collision metric for agents exposed through the interface of this class. Includes collisions with agents not exposed through the interface. Collisions are defined as overlap of agents' bounding boxes, with details determined by the specific method chosen in the config. :param agent_types: An optional list of specific agent types for computing collisions with. Not supported by the collision metrics `nograd` and `nograd-pytorch3d`. :returns: a functor of BxA tensors .. py:class:: Simulator(road_mesh: torchdrivesim.mesh.BirdviewMesh, kinematic_model: Dict[str, torchdrivesim.kinematic.KinematicModel], agent_size: Dict[str, torch.Tensor], initial_present_mask: Dict[str, torch.Tensor], cfg: TorchDriveConfig, renderer: Optional[torchdrivesim.rendering.BirdviewRenderer] = None, lanelet_map: Optional[List[Optional[torchdrivesim.lanelet2.LaneletMap]]] = None, recenter_offset: Optional[torch.Tensor] = None, internal_time: int = 0, traffic_controls: Optional[Dict[str, torchdrivesim.traffic_controls.BaseTrafficControl]] = None, waypoint_goals: Optional[torchdrivesim.goals.WaypointGoal] = None) Bases: :py:obj:`SimulatorInterface` Base simulator, where the agent functor is a dictionary indexed with agent type. :param road_mesh: a mesh indicating the driveable area :param kinematic_model: determines the action space, constraints, and the initial state of all agents :param agent_size: a functor of Bx2 tensors indicating agent length and width :param initial_present_mask: a functor of BxA tensors indicating which agents are initially present and not padding :param cfg: holds various configuration options :param renderer: specify if using a non-standard renderer or static meshes beyond the road mesh (default from config) :param lanelet_map: provide the map to compute orientation losses, one map per batch element where available :param recenter_offset: if the coordinate system from lanelet_map was shifted, this value will be used to shift it back :param internal_time: initial value for step counter :param traffic_controls: applicable traffic controls by type .. py:property:: agent_functor :type: DictAgentTypeFunctor Defines how to apply functions across agent types. .. py:property:: agent_types List of agent types used by this simulator, or `None` if only one type used. .. py:property:: action_size :type: IntPerAgentType Defines the size of the action space for each agent type. .. py:property:: batch_size :type: int .. py:method:: to(device) Modifies the simulator in-place, putting all tensors on the device provided. .. py:method:: copy() Duplicates this simulator, allowing for independent subsequent execution. The copy is relatively shallow, in that the tensors are the same objects but dictionaries referring to them are shallowly copied. .. py:method:: extend(n, in_place=True) Multiplies the first batch dimension by the given number. Like in pytorch3d, this is equivalent to introducing extra batch dimension on the right and then flattening. .. py:method:: select_batch_elements(idx, in_place=True) Picks selected elements of the batch. The input is a tensor of indices into the batch dimension. .. py:method:: validate_agent_types() .. py:method:: validate_tensor_shapes() .. py:method:: validate_agent_count(count_dict) .. py:method:: get_world_center() Returns a Bx2 tensor with the coordinates of the map center. .. py:method:: get_state() Returns a functor of BxAxSt tensors representing current agent states. .. py:method:: get_waypoints() Returns a functor of BxAxMx2 tensors representing current agent waypoints. .. py:method:: get_waypoints_state() Returns a functor of BxAx1 tensors representing current agent waypoints state. .. py:method:: get_waypoints_mask() Returns a functor of BxAxM boolean tensors representing current agent waypoints present mask. .. py:method:: compute_wrong_way() Wrong-way metric for each agent, based on the inner product between the agent and lane direction. See `torchdrivesim.infractions.lanelet_orientation_loss` for details. :returns: a functor of BxA tensors .. py:method:: get_agent_size() Returns a functor of BxAx2 tensors representing agent length and width. .. py:method:: get_agent_type() Returns a functor of BxA long tensors containing agent type indexes relative to the list containing all agent types as returned by `SimulatorInterface.agent_types`. .. py:method:: get_present_mask() Returns a functor of BxA boolean tensors indicating which agents are currently present in the simulation. .. py:method:: get_all_agents_absolute() Returns a functor of BxAx6 tensors, where the last dimension contains the following information: x, y, psi, length, width, present. Typically used to implement non-visual observation modalities. .. py:method:: get_all_agents_relative(exclude_self=True) Returns a functor of BxAxAllx6 tensors, specifying for each of A agents the relative position about the other agents. 'All' is the number of all agents in the simulation, including hidden ones, across all agent types. If `exclude_self` is set, for each agent in A, that agent itself is removed from All. The final dimension has the same meaning as in `get_all_agents_absolute`, except now the positions and orientations are relative to the specified agent. .. py:method:: get_innermost_simulator() -> typing_extensions.Self Returns the innermost concrete Simulator object. The type signature is misleading due to Python limitations. .. py:method:: step(agent_action) Runs the simulation for one step with given agent actions. Input is a functor of BxAxAc tensors, where Ac is determined by the kinematic model. .. py:method:: set_state(agent_state, mask=None) Arbitrarily set the state of the agents, without advancing the simulation. The change is effective immediately, without waiting for the next step. :param agent_state: a functor of BxAx4 tensors with agent states :param mask: a functor of BxA boolean tensors, deciding which agent states to update; all by default .. py:method:: update_present_mask(present_mask) Sets the present mask of agents to the provided value. :param present_mask: a functor of BxA boolean tensors .. py:method:: fit_action(future_state, current_state=None) Computes an action that would (aproximately) produce the desired state. :param future_state: a functor of BxAx4 tensors defining the desired state :param current_state: if different from the current simulation state, in the same format as future state :returns: a functor of BxAxAc tensors .. py:method:: render(camera_xy, camera_psi, res=None, rendering_mask=None, fov=None, waypoints=None, waypoints_rendering_mask=None) Renders the world from bird's eye view using cameras in given positions. :param camera_xy: BxNx2 tensor of x-y positions for N cameras :param camera_psi: BxNx1 tensor of orientations for N cameras :param res: desired image resolution (only square resolutions are supported; by default use value from config) :param rendering_mask: functor of BxNxA tensors, indicating which agents should be rendered each camera :param fov: the field of view of the resulting image in meters (by default use value from config) :param waypoints: BxNxMx2 tensor of `M` waypoints per camera (x,y) :param waypoints_rendering_mask: BxNxM tensor of `M` waypoint masks per camera, indicating which waypoints should be rendered :returns: BxNxCxHxW tensor of resulting RGB images for each camera .. py:method:: compute_offroad() Offroad metric for each agent, defined as the distance to the road mesh. See `torchdrivesim.infractions.offroad_infraction_loss` for details. :returns: a functor of BxA tensors .. py:method:: _compute_collision_of_single_agent(box, remove_self_overlap=None, agent_types=None) Computes the collision metric for an agent specified as a bounding box. Includes collisions with all agents in the simulation, including the ones not exposed through the interface of this class. Used with `discs` and `iou` metrics. :param box: Bx5 tensor, with the last dimension being (x,y,length,width,psi). :param remove_self_overlap: B boolean tensor, where if the input agent is present in the simulation, set this to subtract self-overlap. By default it is assumed that self overlapping exists and will be removed. :param agent_types: An optional list of specific agent types for computing collisions with. By default all available agent types will be used. :returns: a tensor with a single dimension of B elements .. py:method:: _compute_collision_of_multi_agents(mask=None) Computes the collision metric for selected (default all) agents in the simulation. Includes collisions with all agents in the simulation, including the ones not exposed through the interface of this class. Used with `nograd` and `nograd-pytorch3d` metrics. :param mask: a functor of BxA boolean tensors, indicating for which agents to compute the loss (by default use present mask) :returns: a functor of BxA tensors .. py:class:: SimulatorWrapper(simulator: SimulatorInterface) Bases: :py:obj:`SimulatorInterface` Modifies the behavior of an existing simulator, itself acting like a simulator. This base class simply delegates all method calls to the inner simulator. .. py:property:: agent_functor :type: AgentTypeFunctor Defines how to apply functions across agent types. .. py:property:: agent_types List of agent types used by this simulator, or `None` if only one type used. .. py:property:: action_size :type: IntPerAgentType Defines the size of the action space for each agent type. .. py:property:: batch_size :type: int .. py:method:: to(device) -> typing_extensions.Self Modifies the simulator in-place, putting all tensors on the device provided. .. py:method:: copy() Duplicates this simulator, allowing for independent subsequent execution. The copy is relatively shallow, in that the tensors are the same objects but dictionaries referring to them are shallowly copied. .. py:method:: extend(n, in_place=True) Multiplies the first batch dimension by the given number. Like in pytorch3d, this is equivalent to introducing extra batch dimension on the right and then flattening. .. py:method:: select_batch_elements(idx, in_place=True) Picks selected elements of the batch. The input is a tensor of indices into the batch dimension. .. py:method:: get_world_center() Returns a Bx2 tensor with the coordinates of the map center. .. py:method:: get_state() Returns a functor of BxAxSt tensors representing current agent states. .. py:method:: get_agent_size() Returns a functor of BxAx2 tensors representing agent length and width. .. py:method:: get_agent_type() Returns a functor of BxA long tensors containing agent type indexes relative to the list containing all agent types as returned by `SimulatorInterface.agent_types`. .. py:method:: get_present_mask() Returns a functor of BxA boolean tensors indicating which agents are currently present in the simulation. .. py:method:: get_waypoints() Returns a functor of BxAxMx2 tensors representing current agent waypoints. .. py:method:: get_waypoints_state() Returns a functor of BxAx1 tensors representing current agent waypoints state. .. py:method:: get_waypoints_mask() Returns a functor of BxAxM boolean tensors representing current agent waypoints present mask. .. py:method:: get_all_agents_absolute() Returns a functor of BxAx6 tensors, where the last dimension contains the following information: x, y, psi, length, width, present. Typically used to implement non-visual observation modalities. .. py:method:: get_all_agents_relative(exclude_self=True) Returns a functor of BxAxAllx6 tensors, specifying for each of A agents the relative position about the other agents. 'All' is the number of all agents in the simulation, including hidden ones, across all agent types. If `exclude_self` is set, for each agent in A, that agent itself is removed from All. The final dimension has the same meaning as in `get_all_agents_absolute`, except now the positions and orientations are relative to the specified agent. .. py:method:: get_innermost_simulator() -> Simulator Returns the innermost concrete Simulator object. The type signature is misleading due to Python limitations. .. py:method:: step(*args, **kwargs) Runs the simulation for one step with given agent actions. Input is a functor of BxAxAc tensors, where Ac is determined by the kinematic model. .. py:method:: set_state(*args, **kwargs) Arbitrarily set the state of the agents, without advancing the simulation. The change is effective immediately, without waiting for the next step. :param agent_state: a functor of BxAx4 tensors with agent states :param mask: a functor of BxA boolean tensors, deciding which agent states to update; all by default .. py:method:: update_present_mask(*args, **kwargs) Sets the present mask of agents to the provided value. :param present_mask: a functor of BxA boolean tensors .. py:method:: fit_action(*args, **kwargs) Computes an action that would (aproximately) produce the desired state. :param future_state: a functor of BxAx4 tensors defining the desired state :param current_state: if different from the current simulation state, in the same format as future state :returns: a functor of BxAxAc tensors .. py:method:: compute_offroad() Offroad metric for each agent, defined as the distance to the road mesh. See `torchdrivesim.infractions.offroad_infraction_loss` for details. :returns: a functor of BxA tensors .. py:method:: _compute_collision_of_single_agent(box, remove_self_overlap=None, agent_types=None) Computes the collision metric for an agent specified as a bounding box. Includes collisions with all agents in the simulation, including the ones not exposed through the interface of this class. Used with `discs` and `iou` metrics. :param box: Bx5 tensor, with the last dimension being (x,y,length,width,psi). :param remove_self_overlap: B boolean tensor, where if the input agent is present in the simulation, set this to subtract self-overlap. By default it is assumed that self overlapping exists and will be removed. :param agent_types: An optional list of specific agent types for computing collisions with. By default all available agent types will be used. :returns: a tensor with a single dimension of B elements .. py:method:: _compute_collision_of_multi_agents(mask=None) Computes the collision metric for selected (default all) agents in the simulation. Includes collisions with all agents in the simulation, including the ones not exposed through the interface of this class. Used with `nograd` and `nograd-pytorch3d` metrics. :param mask: a functor of BxA boolean tensors, indicating for which agents to compute the loss (by default use present mask) :returns: a functor of BxA tensors .. py:method:: compute_wrong_way() Wrong-way metric for each agent, based on the inner product between the agent and lane direction. See `torchdrivesim.infractions.lanelet_orientation_loss` for details. :returns: a functor of BxA tensors .. py:method:: render(camera_xy, camera_psi, res=None, rendering_mask=None, fov=None, waypoints=None, waypoints_rendering_mask=None) Renders the world from bird's eye view using cameras in given positions. :param camera_xy: BxNx2 tensor of x-y positions for N cameras :param camera_psi: BxNx1 tensor of orientations for N cameras :param res: desired image resolution (only square resolutions are supported; by default use value from config) :param rendering_mask: functor of BxNxA tensors, indicating which agents should be rendered each camera :param fov: the field of view of the resulting image in meters (by default use value from config) :param waypoints: BxNxMx2 tensor of `M` waypoints per camera (x,y) :param waypoints_rendering_mask: BxNxM tensor of `M` waypoint masks per camera, indicating which waypoints should be rendered :returns: BxNxCxHxW tensor of resulting RGB images for each camera .. py:class:: NPCWrapper(simulator: SimulatorInterface, npc_mask: TensorPerAgentType) Bases: :py:obj:`SimulatorWrapper` Designates a certain subset of agents as non-playable characters (NPCs) and removes them from the simulator interface, although they remain in simulation and can be interacted with. Note that the designation of which agents are NPCs needs to be the same across batch elements. Subclasses implement specific policies controlling NPC behaviors. At a minimum, they should implement either `_get_npc_action` or `_npc_teleport_to`. :param npc_mask: A functor of tensors with a single dimension of size A, indicating which agents to replay. .. py:method:: _update_npc_present_mask() -> TensorPerAgentType Computes updated present masks for NPCs, with arbitrary padding for the remaining agents. By default, leaves present masks unchanged. :returns: a functor of BxA boolean tensors, where A is the number of agents in the inner simulator .. py:method:: _get_npc_action() -> TensorPerAgentType Computes the actions for NPCs, with arbitrary padding for actions of the remaining agents. By default, the actions are all zeros, but subclasses can implement more intelligent behavior. :returns: a functor of BxAxAc tensors, where A is the number of agents in the inner simulator .. py:method:: _npc_teleport_to() -> Optional[TensorPerAgentType] Provides the states to which the NPCs should be set after `step`, with arbitrary padding for the remaining agents. By default, no teleportation is performed, but subclasses may use it instead of, or on top of defining the NPC action. :returns: a functor of BxAxSt tensors, where A is the number of agents in the inner simulator, or `None` if no teleportation is required .. py:method:: to(device) -> typing_extensions.Self Modifies the simulator in-place, putting all tensors on the device provided. .. py:method:: copy() Duplicates this simulator, allowing for independent subsequent execution. The copy is relatively shallow, in that the tensors are the same objects but dictionaries referring to them are shallowly copied. .. py:method:: validate_agent_count(count_dict) .. py:method:: get_state() Returns a functor of BxAxSt tensors representing current agent states. .. py:method:: get_waypoints() Returns a functor of BxAxMx2 tensors representing current agent waypoints. .. py:method:: get_waypoints_state() Returns a functor of BxAx1 tensors representing current agent waypoints state. .. py:method:: get_waypoints_mask() Returns a functor of BxAxM boolean tensors representing current agent waypoints present mask. .. py:method:: get_agent_size() Returns a functor of BxAx2 tensors representing agent length and width. .. py:method:: get_agent_type() Returns a functor of BxA long tensors containing agent type indexes relative to the list containing all agent types as returned by `SimulatorInterface.agent_types`. .. py:method:: get_present_mask() Returns a functor of BxA boolean tensors indicating which agents are currently present in the simulation. .. py:method:: get_all_agents_relative(exclude_self=True) Returns a functor of BxAxAllx6 tensors, specifying for each of A agents the relative position about the other agents. 'All' is the number of all agents in the simulation, including hidden ones, across all agent types. If `exclude_self` is set, for each agent in A, that agent itself is removed from All. The final dimension has the same meaning as in `get_all_agents_absolute`, except now the positions and orientations are relative to the specified agent. .. py:method:: set_state(agent_state, mask=None) Arbitrarily set the state of the agents, without advancing the simulation. The change is effective immediately, without waiting for the next step. :param agent_state: a functor of BxAx4 tensors with agent states :param mask: a functor of BxA boolean tensors, deciding which agent states to update; all by default .. py:method:: step(action) Runs the simulation for one step with given agent actions. Input is a functor of BxAxAc tensors, where Ac is determined by the kinematic model. .. py:method:: update_present_mask(present_mask) Sets the present mask of agents to the provided value. :param present_mask: a functor of BxA boolean tensors .. py:method:: render(camera_xy, camera_psi, res=None, rendering_mask=None, fov=None, waypoints=None, waypoints_rendering_mask=None) Renders the world from bird's eye view using cameras in given positions. :param camera_xy: BxNx2 tensor of x-y positions for N cameras :param camera_psi: BxNx1 tensor of orientations for N cameras :param res: desired image resolution (only square resolutions are supported; by default use value from config) :param rendering_mask: functor of BxNxA tensors, indicating which agents should be rendered each camera :param fov: the field of view of the resulting image in meters (by default use value from config) :param waypoints: BxNxMx2 tensor of `M` waypoints per camera (x,y) :param waypoints_rendering_mask: BxNxM tensor of `M` waypoint masks per camera, indicating which waypoints should be rendered :returns: BxNxCxHxW tensor of resulting RGB images for each camera .. py:method:: fit_action(future_state, current_state=None) Computes an action that would (aproximately) produce the desired state. :param future_state: a functor of BxAx4 tensors defining the desired state :param current_state: if different from the current simulation state, in the same format as future state :returns: a functor of BxAxAc tensors .. py:method:: compute_offroad() Offroad metric for each agent, defined as the distance to the road mesh. See `torchdrivesim.infractions.offroad_infraction_loss` for details. :returns: a functor of BxA tensors .. py:method:: compute_wrong_way() Wrong-way metric for each agent, based on the inner product between the agent and lane direction. See `torchdrivesim.infractions.lanelet_orientation_loss` for details. :returns: a functor of BxA tensors .. py:method:: _compute_collision_of_multi_agents(mask=None) Computes the collision metric for selected (default all) agents in the simulation. Includes collisions with all agents in the simulation, including the ones not exposed through the interface of this class. Used with `nograd` and `nograd-pytorch3d` metrics. :param mask: a functor of BxA boolean tensors, indicating for which agents to compute the loss (by default use present mask) :returns: a functor of BxA tensors .. py:method:: extend_tensor(x, batch_dims) :staticmethod: .. py:class:: HomogeneousWrapper(simulator: SimulatorInterface) Bases: :py:obj:`SimulatorWrapper` Removes distinction between agent types, replacing collections with single tensors concatenated across the agent dimension. Only safe to use when all agents share the same kinematic model. .. py:property:: agent_functor :type: SingletonAgentTypeFunctor Defines how to apply functions across agent types. .. py:property:: agent_types List of agent types used by this simulator, or `None` if only one type used. .. py:property:: action_size :type: IntPerAgentType Defines the size of the action space for each agent type. .. py:method:: agent_concat(d, agent_dim, check_agent_count=True) Concatenates a collection of agent tensors along the agent dimension. .. py:method:: agent_split(x, agent_dim) Reverse of `agent_concat`, splits a tensor into a collection of agent tensors. .. py:method:: get_state() Returns a functor of BxAxSt tensors representing current agent states. .. py:method:: get_agent_size() Returns a functor of BxAx2 tensors representing agent length and width. .. py:method:: get_agent_type() Returns a functor of BxA long tensors containing agent type indexes relative to the list containing all agent types as returned by `SimulatorInterface.agent_types`. .. py:method:: get_present_mask() Returns a functor of BxA boolean tensors indicating which agents are currently present in the simulation. .. py:method:: get_waypoints() Returns a functor of BxAxMx2 tensors representing current agent waypoints. .. py:method:: get_waypoints_state() Returns a functor of BxAx1 tensors representing current agent waypoints state. .. py:method:: get_waypoints_mask() Returns a functor of BxAxM boolean tensors representing current agent waypoints present mask. .. py:method:: get_all_agents_absolute() Returns a functor of BxAx6 tensors, where the last dimension contains the following information: x, y, psi, length, width, present. Typically used to implement non-visual observation modalities. .. py:method:: get_all_agents_relative(exclude_self=True) Returns a functor of BxAxAllx6 tensors, specifying for each of A agents the relative position about the other agents. 'All' is the number of all agents in the simulation, including hidden ones, across all agent types. If `exclude_self` is set, for each agent in A, that agent itself is removed from All. The final dimension has the same meaning as in `get_all_agents_absolute`, except now the positions and orientations are relative to the specified agent. .. py:method:: set_state(agent_state, mask=None) Arbitrarily set the state of the agents, without advancing the simulation. The change is effective immediately, without waiting for the next step. :param agent_state: a functor of BxAx4 tensors with agent states :param mask: a functor of BxA boolean tensors, deciding which agent states to update; all by default .. py:method:: step(action) Runs the simulation for one step with given agent actions. Input is a functor of BxAxAc tensors, where Ac is determined by the kinematic model. .. py:method:: update_present_mask(present_mask) Sets the present mask of agents to the provided value. :param present_mask: a functor of BxA boolean tensors .. py:method:: fit_action(future_state, current_state=None) Computes an action that would (aproximately) produce the desired state. :param future_state: a functor of BxAx4 tensors defining the desired state :param current_state: if different from the current simulation state, in the same format as future state :returns: a functor of BxAxAc tensors .. py:method:: compute_offroad() Offroad metric for each agent, defined as the distance to the road mesh. See `torchdrivesim.infractions.offroad_infraction_loss` for details. :returns: a functor of BxA tensors .. py:method:: compute_wrong_way() Wrong-way metric for each agent, based on the inner product between the agent and lane direction. See `torchdrivesim.infractions.lanelet_orientation_loss` for details. :returns: a functor of BxA tensors .. py:method:: _compute_collision_of_multi_agents(mask=None) Computes the collision metric for selected (default all) agents in the simulation. Includes collisions with all agents in the simulation, including the ones not exposed through the interface of this class. Used with `nograd` and `nograd-pytorch3d` metrics. :param mask: a functor of BxA boolean tensors, indicating for which agents to compute the loss (by default use present mask) :returns: a functor of BxA tensors .. py:method:: render(camera_xy, camera_psi, res=None, rendering_mask=None, fov=None, waypoints=None, waypoints_rendering_mask=None) Renders the world from bird's eye view using cameras in given positions. :param camera_xy: BxNx2 tensor of x-y positions for N cameras :param camera_psi: BxNx1 tensor of orientations for N cameras :param res: desired image resolution (only square resolutions are supported; by default use value from config) :param rendering_mask: functor of BxNxA tensors, indicating which agents should be rendered each camera :param fov: the field of view of the resulting image in meters (by default use value from config) :param waypoints: BxNxMx2 tensor of `M` waypoints per camera (x,y) :param waypoints_rendering_mask: BxNxM tensor of `M` waypoint masks per camera, indicating which waypoints should be rendered :returns: BxNxCxHxW tensor of resulting RGB images for each camera .. py:class:: RecordingWrapper(simulator: SimulatorInterface, record_functions: Dict[str, Callable[[SimulatorInterface], Any]], initial_recording: bool = True) Bases: :py:obj:`SimulatorWrapper` Doesn't modify the behavior of the simulator, but records some information at each time step. :param record_functions: those functions are called on `self` after each `step` and their outputs are recorded :param initial_recording: whether to record values when constructing the object, before the first step .. py:method:: copy() Duplicates this simulator, allowing for independent subsequent execution. The copy is relatively shallow, in that the tensors are the same objects but dictionaries referring to them are shallowly copied. .. py:method:: record() Appends the results based on a current state to records. .. py:method:: get_records() Returns the dictionary of recorded values. Each entry is a list with results from subsequent invocations of `record`. .. py:method:: step(agent_action, record=True) By default performs recording at the end of step, but this can be disabled to manually control when to record. .. py:class:: BirdviewRecordingWrapper(simulator: SimulatorInterface, res: torchdrivesim.utils.Resolution = Resolution(1024, 1024), fov: float = 100, camera_xy: Optional[torch.Tensor] = None, camera_psi: Optional[torch.Tensor] = None, initial_recording: bool = True, to_cpu: bool = False) Bases: :py:obj:`RecordingWrapper` Records a visualization of the simulation state at each step. The visualization is obtained by calling `self.render`. :param simulator: a simulator object to wrap :param res: resolution for the recorded images :param fov: field of view for the recorded images :param camera_xy: a Bx2 tensor of x-y camera positions (world center by default) :param camera_psi: a Bx1 tensor of camera orientations :param initial_recording: whether to record an image before the first step :param to_cpu: whether to move recorded images to CPU memory, releasing GPU memory .. py:method:: to(device) -> typing_extensions.Self Modifies the simulator in-place, putting all tensors on the device provided. .. py:method:: copy() Duplicates this simulator, allowing for independent subsequent execution. The copy is relatively shallow, in that the tensors are the same objects but dictionaries referring to them are shallowly copied. .. py:method:: extend(n, in_place=True) Multiplies the first batch dimension by the given number. Like in pytorch3d, this is equivalent to introducing extra batch dimension on the right and then flattening. .. py:method:: select_batch_elements(idx, in_place=True) Picks selected elements of the batch. The input is a tensor of indices into the batch dimension. .. py:method:: get_birdviews(stack: bool = False) -> Union[torch.Tensor, List[torch.Tensor]] Extracts recorded images. :param stack: whether to concatenate results across time, which is placed after batch dimension :returns: a BxTxCxHxW tensor of RGB images if `stack` is set, a list of BxCxHxW images with the same content otherwise .. py:method:: save_gif(filename: str, batch_index: int = 0, fps: float = 10) -> None Saves a GIF to disk using all birdviews for a selected example from the batch. .. py:method:: save_png(filename: str, frame: int, batch_index: int = 0) -> None Saves the given frame as a PNG to disk. .. py:class:: TrajectoryVisualizationWrapper(simulator: SimulatorInterface, ground_truth: torch.Tensor, predictions: torch.Tensor, gt_present_masks: torch.Tensor, predictions_present_masks: torch.Tensor, trajectory_present_masks: torch.Tensor, res: torchdrivesim.utils.Resolution = Resolution(1024, 1024), fov: float = 100, camera_xy: Optional[torch.Tensor] = None, camera_psi: Optional[torch.Tensor] = None, to_cpu: bool = False) Bases: :py:obj:`BirdviewRecordingWrapper` Records birdviews with additional trajectory visualizations. Those visualizations will be visible to agents relying on rendered observations, so typically this wrapper is used with replay states. The wrapper visualizes one ground truth trajectory and S predictions for each agent. Visualized trajectories should be concatenated across the agent dimension. :param ground_truth: BxAxTxSt tensor with ground truth trajectories for all agents :param predictions: BxSxAxTxSt tensor with predicted trajectories :param gt_present_masks: BxSxAxT boolean tensor, indicating which points in predictions should be visualized :param predictions_present_masks: BxAxT boolean tensor, indicating which points in ground truth should be visualized .. py:class:: SelectiveWrapper(simulator: SimulatorInterface, exposed_agent_limit: int, default_action: TensorPerAgentType) Bases: :py:obj:`SimulatorWrapper` Only exposes a subset of agents from the inner simulator. The selection may change over time, but there is a fixed number of agent slots available. Which slots are actually used is indicated by the presence mask. The base class exposes the first E agents and the children should override `update_exposed_agents` and `get_present_mask` to customize it. Typically, hidden agents are either absent or their state is set externally, but by default the `default_action` is executed for them. :param simulator: existing simulator to wrap :param exposed_agent_limit: denoted as E in comments, sets the number of slots for exposed agents :param default_action: a functor of BxAxAc tensors, defining the default action to use for non-exposed agents .. py:method:: update_exposed_agents() -> None There should always be E exposed agents, although some of them may be marked as absent. .. py:method:: get_exposed_agents() -> TensorPerAgentType :returns: a functor of BxE int tensors of indices of exposed agents .. py:method:: is_exposed() -> TensorPerAgentType :returns: a functor of BxA boolean tensors indicating which agents are exposed .. py:method:: _restrict_tensor(tensor: TensorPerAgentType, agent_dim: int) -> TensorPerAgentType Selects exposed agents from a given tensor, along the supplied agent dimension. .. py:method:: _extend_tensor(tensor: TensorPerAgentType, padding: TensorPerAgentType) -> TensorPerAgentType Given a tensor of exposed agents, constructs a tensor of all agents, using supplied padding. Agent dimension should be the penultimate one. .. py:method:: to(device) -> typing_extensions.Self Modifies the simulator in-place, putting all tensors on the device provided. .. py:method:: copy() Duplicates this simulator, allowing for independent subsequent execution. The copy is relatively shallow, in that the tensors are the same objects but dictionaries referring to them are shallowly copied. .. py:method:: extend(n, in_place=True) Multiplies the first batch dimension by the given number. Like in pytorch3d, this is equivalent to introducing extra batch dimension on the right and then flattening. .. py:method:: select_batch_elements(idx, in_place=True) Picks selected elements of the batch. The input is a tensor of indices into the batch dimension. .. py:method:: get_state() Returns a functor of BxAxSt tensors representing current agent states. .. py:method:: get_present_mask() Returns a functor of BxA boolean tensors indicating which agents are currently present in the simulation. .. py:method:: get_agent_size() Returns a functor of BxAx2 tensors representing agent length and width. .. py:method:: get_agent_type() Returns a functor of BxA long tensors containing agent type indexes relative to the list containing all agent types as returned by `SimulatorInterface.agent_types`. .. py:method:: get_waypoints() Returns a functor of BxAxMx2 tensors representing current agent waypoints. .. py:method:: get_waypoints_state() Returns a functor of BxAx1 tensors representing current agent waypoints state. .. py:method:: get_waypoints_mask() Returns a functor of BxAxM boolean tensors representing current agent waypoints present mask. .. py:method:: get_all_agents_absolute() Returns a functor of BxAx6 tensors, where the last dimension contains the following information: x, y, psi, length, width, present. Typically used to implement non-visual observation modalities. .. py:method:: get_all_agents_relative(exclude_self=True) Returns a functor of BxAxAllx6 tensors, specifying for each of A agents the relative position about the other agents. 'All' is the number of all agents in the simulation, including hidden ones, across all agent types. If `exclude_self` is set, for each agent in A, that agent itself is removed from All. The final dimension has the same meaning as in `get_all_agents_absolute`, except now the positions and orientations are relative to the specified agent. .. py:method:: set_state(agent_state, mask=None) Arbitrarily set the state of the agents, without advancing the simulation. The change is effective immediately, without waiting for the next step. :param agent_state: a functor of BxAx4 tensors with agent states :param mask: a functor of BxA boolean tensors, deciding which agent states to update; all by default .. py:method:: fit_action(future_state, current_state=None) Computes an action that would (aproximately) produce the desired state. :param future_state: a functor of BxAx4 tensors defining the desired state :param current_state: if different from the current simulation state, in the same format as future state :returns: a functor of BxAxAc tensors .. py:method:: render(camera_xy, camera_psi, res=None, rendering_mask=None, fov=None, waypoints=None, waypoints_rendering_mask=None) Renders the world from bird's eye view using cameras in given positions. :param camera_xy: BxNx2 tensor of x-y positions for N cameras :param camera_psi: BxNx1 tensor of orientations for N cameras :param res: desired image resolution (only square resolutions are supported; by default use value from config) :param rendering_mask: functor of BxNxA tensors, indicating which agents should be rendered each camera :param fov: the field of view of the resulting image in meters (by default use value from config) :param waypoints: BxNxMx2 tensor of `M` waypoints per camera (x,y) :param waypoints_rendering_mask: BxNxM tensor of `M` waypoint masks per camera, indicating which waypoints should be rendered :returns: BxNxCxHxW tensor of resulting RGB images for each camera .. py:method:: step(action) Runs the simulation for one step with given agent actions. Input is a functor of BxAxAc tensors, where Ac is determined by the kinematic model. .. py:method:: update_present_mask(*args, **kwargs) :abstractmethod: Sets the present mask of agents to the provided value. :param present_mask: a functor of BxA boolean tensors .. py:method:: _compute_collision_of_multi_agents(mask=None) Computes the collision metric for selected (default all) agents in the simulation. Includes collisions with all agents in the simulation, including the ones not exposed through the interface of this class. Used with `nograd` and `nograd-pytorch3d` metrics. :param mask: a functor of BxA boolean tensors, indicating for which agents to compute the loss (by default use present mask) :returns: a functor of BxA tensors .. py:class:: BoundedRegionWrapper(simulator: SimulatorInterface, exposed_agent_limit: int, default_action: TensorPerAgentType, warmup_timesteps: int, cutoff_polygon_verts: TensorPerAgentType) Bases: :py:obj:`SelectiveWrapper` A variant of `SelectiveWrapper` that exposes agents contained within a given polygon defining the area of interest. It additionally provides a method to determine which of the exposed agents have been "warmed-up", which means that they have been exposed for a certain number of time steps, but that is separate from the main functionality. :param warmup_timesteps: after how many steps of being exposed are the agents considered "warmed-up" :param cutoff_polygon_verts: vertices defining the bounding convex polygon, in either clockwise or counter-closkwise order, provided as a functor of BxNx2 tensors, so that each agent type can use a different polygon .. py:method:: update_exposed_agents() There should always be E exposed agents, although some of them may be marked as absent. .. py:method:: get_present_mask() Returns a functor of BxA boolean tensors indicating which agents are currently present in the simulation. .. py:method:: is_warmed_up() :returns: a functor of BxA boolean tensors indicating which agents are warmed-up .. py:method:: to(device) -> typing_extensions.Self Modifies the simulator in-place, putting all tensors on the device provided. .. py:method:: copy() Duplicates this simulator, allowing for independent subsequent execution. The copy is relatively shallow, in that the tensors are the same objects but dictionaries referring to them are shallowly copied. .. py:method:: extend(n, in_place=True) Multiplies the first batch dimension by the given number. Like in pytorch3d, this is equivalent to introducing extra batch dimension on the right and then flattening. .. py:method:: select_batch_elements(idx, in_place=True) Picks selected elements of the batch. The input is a tensor of indices into the batch dimension. .. py:class:: NoReentryBoundedRegionWrapper(simulator: SimulatorInterface, exposed_agent_limit, default_action, warmup_timesteps, cutoff_polygon_verts: Optional[TensorPerAgentType] = None) Bases: :py:obj:`BoundedRegionWrapper` A variant of `BoundedRegionWrapper` that does not allow reentry. This means that if an agent is not exposed at a certain time, it will never be exposed in the future. However, the agents that can stop being exposed if they exit the area of interest. .. py:method:: to(device) -> typing_extensions.Self Modifies the simulator in-place, putting all tensors on the device provided. .. py:method:: extend(n, in_place=True) Multiplies the first batch dimension by the given number. Like in pytorch3d, this is equivalent to introducing extra batch dimension on the right and then flattening. .. py:method:: select_batch_elements(idx, in_place=True) Picks selected elements of the batch. The input is a tensor of indices into the batch dimension. .. py:method:: update_exposed_agents() There should always be E exposed agents, although some of them may be marked as absent. .. py:method:: update_present_mask(present_mask) Sets the present mask of agents to the provided value. :param present_mask: a functor of BxA boolean tensors .. py:method:: get_present_mask() Returns a functor of BxA boolean tensors indicating which agents are currently present in the simulation.