torchdrivesim.rendering ======================= .. py:module:: torchdrivesim.rendering .. autoapi-nested-parse:: Renderers used to visualize the state of the environments. Currently three backends are supported (opencv, pytorch3d and nvdiffrast), along with a dummy renderer generating black images. Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/torchdrivesim/rendering/base/index /autoapi/torchdrivesim/rendering/cv2/index /autoapi/torchdrivesim/rendering/nvdiffrast/index /autoapi/torchdrivesim/rendering/pytorch3d/index Attributes ---------- .. autoapisummary:: torchdrivesim.rendering.logger Classes ------- .. autoapisummary:: torchdrivesim.rendering.RendererConfig torchdrivesim.rendering.DummyRendererConfig torchdrivesim.rendering.BirdviewRenderer torchdrivesim.rendering.DummyRenderer torchdrivesim.rendering.CV2RendererConfig torchdrivesim.rendering.CV2Renderer torchdrivesim.rendering.Pytorch3DRendererConfig torchdrivesim.rendering.Pytorch3DRenderer torchdrivesim.rendering.NvdiffrastRendererConfig torchdrivesim.rendering.NvdiffrastRenderer Functions --------- .. autoapisummary:: torchdrivesim.rendering.renderer_from_config Package Contents ---------------- .. py:class:: RendererConfig Determines behavior of the renderer. Subclasses determine renderer class used. .. py:attribute:: backend :type: str :value: 'default' .. py:attribute:: render_agent_direction :type: bool :value: True .. py:attribute:: left_handed_coordinates :type: bool :value: False .. py:attribute:: highlight_ego_vehicle :type: bool :value: False .. py:class:: DummyRendererConfig Bases: :py:obj:`RendererConfig` For DummyRenderer. .. py:attribute:: backend :type: str :value: 'dummy' .. py:class:: BirdviewRenderer(cfg: RendererConfig, device: Optional[torch.device] = None, batch_size: Optional[int] = None, static_mesh: Optional[torchdrivesim.mesh.BirdviewMesh] = None, world_center: Optional[torch.Tensor] = None, color_map: Optional[Dict[str, Tuple[int, int, int]]] = None, rendering_levels: Optional[Dict[str, float]] = None, res: torchdrivesim.utils.Resolution = Resolution(64, 64), fov: float = 35) Bases: :py:obj:`abc.ABC` A renderer producing simple 2D birdview images based on static background meshes and rectangular agents. Currently only square resolutions are supported. The renderer always operates in batch mode, with a single batch dimension on the left. :param cfg: configuration object, usually subclassed :param device: torch device used for rendering :param batch_size: if road_mesh is not specified, this is used to determine batch size :param static_mesh: BirdviewMesh object specifying drivable surface (empty mesh is used if not provided) :param world_center: Bx2 float tensor, defaults to geometric centre of the road mesh :param color_map: a dictionary of RGB tuples in 0-255 range specifying colors of different rendered elements :param res: default resolution .. py:method:: get_color(element_type: str) -> Tuple[int, int, int] .. py:method:: to(device: torch.device) Moves the renderer to another device in place. .. py:method:: add_static_meshes(meshes: List[torchdrivesim.mesh.BirdviewMesh]) -> None Includes additional static elements to render. .. py:method:: copy() .. py:method:: expand(n: int) Adds another dimension with size n on the right of the batch dimension and flattens them. Returns a new renderer, without modifying the current one. .. py:method:: select_batch_elements(idx: torch.Tensor) Selects given elements from the batch, potentially with repetitions. Returns a new renderer, without modifying the current one. :param idx: one-dimensional integer tensor .. py:method:: render_static_meshes(camera_xy: Optional[torch.Tensor] = None, camera_sc: Optional[torch.Tensor] = None, res: torchdrivesim.utils.Resolution = None, fov: float = None) -> torch.Tensor Render a single birdview of the static mesh only. Nc is the number of cameras. C=3 is the number of RGB channels. :param camera_xy: Ncx2 tensor of camera positions :param camera_sc: Ncx2 tensor of camera orientations (sine and cosine of yaw angle) :param res: Resolution, currently only square resolutions are supported :param fov: Field of view in meters :returns: birdview image tensor of shape NcxHxWxC .. py:method:: transform(points: torch.Tensor, pose: torch.Tensor) -> torch.Tensor Given points relative to a pose, produce absolute positions of the points. There can be zero or more batch dimensions. :param points: BxNx2 tensor :param pose: Bx3 tensor of position (x,y) and orientation (yaw angle in radians) :returns: Bx2 tensor of absolute positions .. py:method:: make_actor_mesh(agent_state: Dict[str, torch.Tensor], agent_attributes: Dict[str, torch.Tensor]) -> torchdrivesim.mesh.BirdviewMesh Creates a mesh representing given actors. Each vertex and each face corresponds to a unique agent and both vertices and faces for each agent are continuous in the resulting tensor to allow for subsequent masking. For each agent there are seven vertices and three faces, specifying its bounding box and direction. Direction vertices use the 'direction' category, while agent categories are copied from input dictionaries. .. py:method:: render_frame(agent_state: Dict[str, torch.Tensor], agent_attributes: Dict[str, torch.Tensor], camera_xy: Optional[torch.Tensor] = None, camera_sc: Optional[torch.Tensor] = None, rendering_mask: Dict[str, torch.Tensor] = None, res: Optional[torchdrivesim.utils.Resolution] = None, traffic_controls: Optional[Dict[str, torchdrivesim.traffic_controls.BaseTrafficControl]] = None, fov: Optional[float] = None, waypoints: Optional[torch.Tensor] = None, waypoints_rendering_mask: Optional[torch.Tensor] = None) -> torch.Tensor Renders the agents and traffic controls on top of the static mesh. Cameras batch size is (B*Nc), which corresponds to using Nc cameras per batch element. This extra dimension is added on the right of batch dimension and flattened, to match the semantics of `extend` from pytorch3d. If cameras is None, one egocentric camera per agent is used, that is Nc = A. :param agent_state: maps agent types to state tensors of shape BxAxSt, where St >= 3 and the first three components are x coordinate, y coordinate, and orientation in radians :param agent_attributes: maps agent types to static attributes tensors of shape BxAxAttr, where Attr >= 2 and the first two components are length and width of the agent :param camera_xy: BxNcx2 tensor of camera positions, by default one camera placed on each agent :param camera_sc: BxNcx2 tensor of camera orientations (sine and cosine), by default matching agent orientations :param rendering_mask: BxNcxA tensor per agent type, indicating which cameras see which agents :param res: resolution HxW of the resulting image, currently only square resolutions are supported :param traffic_controls: traffic controls by type (traffic_light, yield, etc.) :param fov: Field of view in meters :param waypoints: BxNcxMx2 tensor of `M` waypoints per camera (x,y) :param waypoints_rendering_mask: BxNcxM tensor of `M` waypoint masks per camera, indicating which waypoints should be rendered :returns: tensor image of float RGB values in [0,255] range with shape shape (B*Nc)xAxCxHxW .. py:method:: render_mesh(mesh: torchdrivesim.mesh.BirdviewMesh, res: torchdrivesim.utils.Resolution, cameras: Cameras) -> torch.Tensor :abstractmethod: Renders a given mesh, producing BxHxWxC tensor image of float RGB values in [0,255] range. .. py:method:: construct_cameras(xy: torch.Tensor, sc: torch.Tensor, scale: Optional[float] = None) -> Cameras Create PyTorch3D cameras object for given positions and orientations. Input tensor dimensions should be Bx2. .. py:method:: build_verts_faces_from_bounding_box(bbs: torch.Tensor, z: float = 2) -> Tuple[torch.Tensor, torch.Tensor] Triangulates actors for rendering. Input is a tensor of bounding boxes of shape ...xAx4x2, where A is the number of actors. Outputs are shaped ...x4*Ax2 and ...x2*Ax3 respectively. .. py:method:: make_traffic_controls_mesh(traffic_controls: Dict[str, torchdrivesim.traffic_controls.BaseTrafficControl]) -> torchdrivesim.mesh.BirdviewMesh Create a mesh showing traffic controls. .. py:method:: make_direction_mesh(lenwid: torch.Tensor, pose: torch.Tensor, size: float = 0.3) -> torchdrivesim.mesh.BaseMesh Create a mesh indicating the direction of each agent. :param lenwid: BxAx2 tensor specifying length and width of the agents :param pose: Bx3 tensor of position (x,y) and orientation (yaw angle in radians) :param size: determines the size of the triangle indicating direction .. py:method:: make_waypoint_mesh(waypoints: torch.Tensor, radius: float = 2.0, num_triangles: int = 10) -> torchdrivesim.mesh.BirdviewMesh Create a mesh of the given waypoints. :param waypoints: BxNcxMx3 tensor of `M` waypoints per camera (x,y,psi) :param radius: float radius of the disc :param num_triangles: int number of triangles used for the disc .. py:class:: DummyRenderer(cfg: RendererConfig, device: Optional[torch.device] = None, batch_size: Optional[int] = None, static_mesh: Optional[torchdrivesim.mesh.BirdviewMesh] = None, world_center: Optional[torch.Tensor] = None, color_map: Optional[Dict[str, Tuple[int, int, int]]] = None, rendering_levels: Optional[Dict[str, float]] = None, res: torchdrivesim.utils.Resolution = Resolution(64, 64), fov: float = 35) Bases: :py:obj:`BirdviewRenderer` Produces a black image of the required size. Mostly used for debugging and benchmarking. .. py:method:: render_mesh(mesh: torchdrivesim.mesh.BirdviewMesh, res: torchdrivesim.utils.Resolution, cameras: Cameras) -> torch.Tensor Renders a given mesh, producing BxHxWxC tensor image of float RGB values in [0,255] range. .. py:class:: CV2RendererConfig Bases: :py:obj:`torchdrivesim.rendering.RendererConfig` Determines behavior of the renderer. Subclasses determine renderer class used. .. py:attribute:: backend :type: str :value: 'cv2' .. py:attribute:: trim_mesh_before_rendering :type: bool :value: True .. py:class:: CV2Renderer(cfg: CV2RendererConfig, *args, **kwargs) Bases: :py:obj:`torchdrivesim.rendering.BirdviewRenderer` Renderer based on OpenCV. Slow, but easy to install. Renders on CPU. .. py:method:: render_mesh(mesh: torchdrivesim.mesh.BirdviewMesh, res: torchdrivesim.utils.Resolution, cameras: torchdrivesim.rendering.base.Cameras) -> torch.Tensor Renders a given mesh, producing BxHxWxC tensor image of float RGB values in [0,255] range. .. py:class:: Pytorch3DRendererConfig Bases: :py:obj:`torchdrivesim.rendering.base.RendererConfig` Configuration of pytorch3d-based renderer. .. py:attribute:: backend :type: str :value: 'pytorch3d' .. py:attribute:: differentiable_rendering :type: RenderingBlend .. py:class:: Pytorch3DRenderer(cfg: Pytorch3DRendererConfig, *args, **kwargs) Bases: :py:obj:`torchdrivesim.rendering.base.BirdviewRenderer` Renderer based on pytorch3d, using an orthographic projection and a trivial shader. Works on both GPU and CPU, but CPU is very slow. .. py:method:: render_mesh(mesh: torchdrivesim.mesh.BirdviewMesh, res: torchdrivesim.utils.Resolution, cameras: torchdrivesim.rendering.base.Cameras) -> torch.Tensor Renders a given mesh, producing BxHxWxC tensor image of float RGB values in [0,255] range. .. py:method:: make_renderer(res, blend, background_color) :classmethod: .. py:class:: NvdiffrastRendererConfig Bases: :py:obj:`torchdrivesim.rendering.base.RendererConfig` Configuration of nvdiffrast-based renderer. .. py:attribute:: backend :type: str :value: 'nvdiffrast' .. py:attribute:: antialias :type: bool :value: False .. py:attribute:: opengl :type: bool :value: False .. py:attribute:: max_minibatch_size :type: Optional[int] :value: None .. py:class:: NvdiffrastRenderer(cfg: NvdiffrastRendererConfig, *args, **kwargs) Bases: :py:obj:`torchdrivesim.rendering.base.BirdviewRenderer` Similar to PyTorch3DRenderer, and producing indistinguishable images, but sometimes faster. Note that nvdiffrast requires separate installation and is subject to its own license terms. .. py:method:: render_mesh(mesh: torchdrivesim.mesh.BirdviewMesh, res: torchdrivesim.utils.Resolution, cameras: torchdrivesim.rendering.base.Cameras) -> torch.Tensor Renders a given mesh, producing BxHxWxC tensor image of float RGB values in [0,255] range. .. py:data:: logger .. py:function:: renderer_from_config(cfg: base.RendererConfig, *args, **kwargs) -> base.BirdviewRenderer Construct the selected renderer from config, by default using Pytorch3DRenderer. Additional arguments are passed to the constructor.