OSCR

Brain-inspired spatial intelligence for embodied agents.

Code ↔ Paper

16 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 16 matches · 2 of them tie a paragraph to a whole file, not to given lines: weak matches, whose lines are not tinted
  1. [1] § Methods › Experimental details › Simulator and datasets ↔ LLMAgent.py, lines 145–204 · score 0.71 · extrinsic attributes, intrinsic attributes, surrounding environmental, language, scenes
  2. [2] § Methods › The BSC-Nav framework › Working memory ↔ LLMAgent.py, lines 208–270 · score 0.70 · kitchen island, confidence scores, target locations, prompts, GPT, matching
  3. [3] § Results › Universal navigation across modalities and granularities ↔ demo.py, lines 25–147 · score 0.67 · navigation trajectories, mp3d, hm3d, navigation tasks, benchmark, episodes
  4. [4] § Methods › Experimental details › Simulator and datasets ↔ textnav_benchmark.py, lines 89–156 · score 0.65 · extrinsic attributes, intrinsic attributes, hm3d, episodes, navigation, scenes
  5. [5] § Methods › Experimental details › Simulation environments ↔ demo.py, lines 25–147 · score 0.63 · Stable Diffusion, success distance, working memory, Medium, exploration, benchmark
  6. [6] § Methods › The BSC-Nav framework › Cognitive map ↔ GES_vlnce/memory.py, lines 834–895 · score 0.62 · radial distance, patch tokens, weight, height, depth, RGB
  7. [7] § Methods › The BSC-Nav framework › Cognitive map ↔ memory_2.py, lines 842–903 · score 0.62 · radial distance, patch tokens, weight, height, depth, RGB
  8. [8] § Results › Construction and exploitation of structured spatial memory in embodied agents ↔ LLMAgent.py, lines 208–270 · score 0.62 · confidence score, highest confidence, target locations, answering, semantic, memory
  9. [9] § Methods › Experimental details › Simulation environments ↔ args.py, the whole file · a weak match · score 0.60 · Stable Diffusion, success distance, working memory, Medium, benchmark, depth
  10. [10] § Methods › Experimental details › Real-world deployment ↔ habitat-lab/habitat/datasets/rearrange/navmesh_utils.py, lines 592–640 · score 0.58 · angular speed, linear speed, threshold, robot, configured, position
  11. [11] § Methods › Experimental details › Foundation model specifications ↔ GES_vlnce/memory.py, lines 39–144 · score 0.56 · YOLO World, WorldV2, quantized, patch, Diffusion, tokens
  12. [12] § Methods › Experimental details › Foundation model specifications ↔ memory_2.py, lines 39–145 · score 0.56 · YOLO World, WorldV2, quantized, patch, Diffusion, tokens
  13. [13] § Methods › The BSC-Nav framework › Low-level navigation policy generation ↔ src/esp/nav/GreedyFollower.h, lines 17–163 · score 0.56 · avoid obstacles, greedy, planner, shortest, sequence, motion
  14. [14] § Methods › Experimental details › Simulator and datasets ↔ args.py, the whole file · a weak match · score 0.55 · HM3D scenes, MP3D scenes, benchmarks, episodes, navigation
  15. [15] § Methods › The BSC-Nav framework › Working memory ↔ LLMAgent.py, lines 9–67 · score 0.54 · enriched descriptions, refine, texture, GPT, model
  16. [16] § Methods › Experimental details › Baseline methods and comparison ↔ GES_vlnce/VLN_CE/vlnce_baselines/models/seq2seq_policy.py, lines 52–179 · score 0.51 · action embeddings, networks, vision, encoding, encoders, sequences

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 991 lines · 47 KB · no license · 4 matches

  1. from openai import OpenAI
  2. import json
  3. from io import BytesIO
  4. import base64
  5. from PIL import Image
  6. # from qwen_vl_utils import process_vision_info
  9. def imagenary_helper_visaug(client, text_prompt, vis):
  10. base64_images = image_to_base64(vis)
  11. completion = client.chat.completions.create(
  12. model="gpt-4o",
  13. timeout=500,
  14. messages=[
  15. {"role": "system", "content": "You are a helpful assistant."},
  16. {"role": "user", "content": [
  17. {"type": "text", "text": """
  18. You are an expert in generating prompts for text-to-image models.
  19. Your task is to enhance a given original goal description, which often only mentions a general category or a simple phrase describing the target object, by incorporating detailed context from four current scene images. You will create a more imaginative and contextually enriched description, using elements observed in the scene to create a coherent and vivid visual. This description will be used to guide a text-to-image model to generate an image that aligns with the style and context of the current scene.
  20. It is crucial that the target object remains the primary visual focus of the image. To complete this task effectively, it's suggested to follow these steps:
  21. 1. **Understand the Environment**: Extract and comprehend details from the provided observation images, such as the overall style, decoration, and elements of the scene. For example, is this a modern home, a classical residence, an exhibition hall, or an office? Consider the style in detail, and understand the context of the environment.
  22. 2. **Expand the Original Description **: Based on the scene analysis from step 1, enrich the original description with finer details. This may include materials, colors, textures, placement, and environmental elements surrounding the target object.
  23. 3. **Maintain Visual Focus **: Ensure that any additional context or background details do not overshadow the main target object. The primary subject should remain the focal point of the generated image, using language that emphasizes its prominence in the scene.
  25. **Guidelines for Creating Enhanced Descriptions: **
  26. 1. Details: Include sensory details like colors, textures, lighting, and reflections.
  27. 2. Background Elements: Add appropriate background elements that complement the scene without detracting from the focus of the image.
  28. Focus Phrasing: Use language that naturally draws attention to the target object (e.g., "centered," "prominently placed," "as the focal point").
  29. 3. Balance: Strike a balance between richness and simplicity. The target object or scene should always dominate the final image.
  30. **Enhanced Description Output Requirements: **
  31. 1. Provide a refined and detailed description in English.
  32. 2. Ensure the enhancement creates a vivid, coherent, and engaging scene, supporting the original description.
  33. 3. Avoid overly complex narratives or elements that distract from the primary object or scene.
  34. 4. Keep the description concise, limiting it to 70 words or less.
  36. **Examples:**
  37. 1.
  38. Original Goal Description: A green vase.
  39. Enhanced Description: A vibrant green ceramic vase, with a glossy, smooth surface, placed centrally on a polished wooden table. Soft natural light illuminates the vase from the large window behind it, casting gentle shadows on the table. The surrounding room is decorated in minimalist modern style with neutral tones, ensuring the vase is the central focal point of the scene.
  40. 2.
  41. Original Goal Description: A armchair.
  42. Enhanced Description: A sleek, modern blue armchair, upholstered in soft velvet, positioned prominently in a stylish living room. The chair is placed near a large floor-to-ceiling window, allowing natural light to highlight its deep blue hue. The room features minimalist decor with white walls, light wood flooring, and a few abstract art pieces on the walls. The armchair stands out as the main focal point, inviting comfort and relaxation.
  43. 3.
  44. Original Goal Description: A desk.
  45. Enhanced Description: A robust metal desk, with a weathered, matte surface, placed against a brick wall in an industrial-style office. The desk features exposed steel legs, and on its surface lies a sleek laptop, a coffee mug, and a few scattered papers. Overhead, a vintage filament bulb hangs from a chain, casting a warm glow over the scene. The surrounding decor includes minimalistic shelves and a large plant in the corner, yet the desk remains the focal point in the room’s urban, raw atmosphere.
  47. """
  48. },
  49. {"type": "text", "text": f"""
  50. Now, the Original Goal Description is "{text_prompt}", and the observation images are:
  51. """
  52. },
  53. {"type": "text", "text": f"observation1:"},
  54. {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_images[0]}"}},
  55. {"type": "text", "text": f"observation2:"},
  56. {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_images[2]}"}},
  57. {"type": "text", "text": f"""
  58. please follow the above requirements and examples to enhance this description, think step by step and give your analysis process and the final enhancement description following the format:
  59. **analysis process**: [your analysis process here]
  60. **enhancement description**: [your enhancement description here]
  61. """
  62. }]
  63. }]
  65. )
  67. return completion.choices[0].message.content
  70. def imagenary_helper(client, text_prompt):
  71. completion = client.chat.completions.create(
  72. model="gpt-4o",
  73. timeout=500,
  74. messages=[
  75. {"role": "system", "content": "You are a helpful assistant."},
  76. {"role": "user", "content": f"""
  77. You are an expert in refining and elaborating simple scene or object descriptions for use in text-to-image generation.
  78. Your task is to take a given raw navigation target description—often just a short phrase mentioning an object or a simple scene—and enhance it by adding imaginative and contextual details.
  79. It is crucial that the original object(s) remain the visual focal point. To achieve this:
  81. 1.**Expand on the Original Description**: Add details about the object(s), their materials, colors, textures, and immediate surroundings.
  82. 2.**Maintain Visual Dominance**: Ensure that any additional context or setting you provide does not overshadow the main elements described in the original text.
  83. The main subjects should remain the most prominent features of the resulting image.
  84. 3.**Emphasize the Main Objects**: Use language that clearly highlights the original objects, ensuring they stand out as the central focus.
  86. **Output Requirements:**
  87. 1.Provide a refined, detailed description in English.
  88. 2.Ensure the enhancements create a vivid, coherent, and inviting scene that supports the original description.
  89. 3.Avoid overly intricate storytelling or elements that distract from the original object or scene.
  90. 4.Please avoid overly long descriptions, limit descriptions to 70 words or less.
  92. **Guidelines for Creating Enhanced Descriptions:**
  94. 1.**Detailing**: Incorporate sensory details such as color, texture, lighting, and reflections.
  95. 2.**Contextual Elements**: Add subtle background elements or environmental details that complement but do not compete with the main subject.
  96. 3.**Focus Phrasing**: Use language that naturally draws attention to the original object(s) (e.g., “centered,” “prominently placed,” “serving as the focal point”).
  97. 4.**Balance**: Maintain a balance between enrichment and simplicity. The primary object or scene described in the original prompt should always dominate the final image.
  98. """
  99. },
  100. {"role": "user", "content": f"""
  102. **Examples**:
  103. 1.
  104. Original Description:
  105. a TV screen above cabinets.
  106. Enhanced Description:
  107. A sleek, flat-screen TV mounted above a set of smooth, white cabinetry. The TV's reflective surface catches the soft glow of recessed lighting,
  108. while the clean, minimalist cabinets provide a neat base that keeps the television as the primary visual anchor.
  109. 2.
  110. Original Description:
  111. a marble island in kitchen.
  112. Enhanced Description:
  113. A polished white marble island centered in a modern kitchen. Delicate gray veining runs across its surface, subtly reflecting under the warm overhead lights.
  114. Simple barstools and neat, neutral-toned countertops frame the island, ensuring it remains the kitchen's focal point.
  115. 3.
  116. Original Description:
  117. a coffee mug on a desk.
  118. Enhanced Description:
  119. A sturdy ceramic coffee mug, ivory in color, resting on a clean, wooden desk. Soft light from a nearby window gently highlights the mug's curved handle and the steam rising from its freshly poured contents.
  120. A simple laptop and a neatly arranged notepad stay in the background, making the mug stand out as the central feature.
  121. 4.
  122. Original Description:
  123. lamp.
  124. Enhanced Description:
  125. A classic white table lamp with a slender stem and an elegant lampshade, standing on a minimalist nightstand. The soft glow from the lamp illuminates its graceful lines,
  126. while the uncluttered background ensures the lamp remains the central feature of the scene.
  127. 5.
  128. Original Description:
  129. a painting.
  130. Enhanced Description:
  131. A colorful, abstract painting mounted on a textured, red-brick wall. The painting's bold brushstrokes and vibrant hues stand out sharply against the wall's rough surface.
  132. Soft track lighting above gently illuminates the artwork, drawing the eye directly to it.
  134. """
  135. },
  136. {"role": "user", "content": f"""
  137. Now, the original description is "{text_prompt}", please follow the above requirements and examples to enhance this description, and directly output the enhanced description.
  138. """
  139. },
  141. ])
  143. return completion.choices[0].message.content
  145. def imagenary_helper_long_text(client, text_prompt):
  146. goal_text_intrinsic, goal_text_extrinsic = text_prompt[0], text_prompt[1]
  147. completion = client.chat.completions.create(
  148. model="gpt-4o",
  149. timeout=500,
  150. messages=[
  151. {"role": "system", "content": "You are a helpful assistant."},
  152. {"role": "user", "content": f"""
  153. You are an expert in refining and elaborating simple scene or object descriptions for use in text-to-image generation.
  155. Your task is to reasonably merge the text description of the object instance and the text description of the surrounding environment, and output a complete description text to guide a text-to-image model to generate a photo of the object. It is crucial that the instance object(s) remain the visual focal point. To achieve these, you should:
  157. 1.**Expand on the Original Description**: Based on the original object description and environment description, you should provides a more fine-grained description of materials, colors, and textures.
  159. 2.**Maintain Visual Dominance**: Ensure that any additional context or setting you provide does not overshadow the main elements described in the original text. the main object should remain the most prominent features of the resulting image.
  161. **Output Requirements:**
  163. 1.Provide a refined, detailed description in English.
  165. 2.Ensure the enhancements create a vivid, coherent, and inviting scene that supports the original description.
  167. 3.Avoid overly intricate storytelling or elements that distract from the original object or scene.
  169. 4.The extended description must follow the original description and not contradict it, such as color, material, and appearance.
  171. 4.Please avoid overly long descriptions, limit descriptions to 70 words or less.
  173. **Guidelines for Creating Enhanced Descriptions:**
  175. 1.**Detailing**: Incorporate sensory details such as color, texture, lighting, and reflections.
  177. 2.**Contextual Elements**: Add subtle background elements or environmental details that complement but do not compete with the main subject.
  179. 3.**Focus Phrasing**: Use language that naturally draws attention to the original object(s) (e.g., “centered,” “prominently placed,” “serving as the focal point”).
  181. 4.**Balance**: Maintain a balance between enrichment and simplicity. The primary object or scene described in the original prompt should always dominate the final image.
  182. """
  183. },
  184. {"role": "user", "content": f"""
  186. **Examples**:
  188. "intrinsic_attributes": "According to the image description, this chair is an old wooden structure with blue seat cushions. Its specific material and color cannot be determined because there are no other details provided about its body or frame."
  190. "extrinsic_attributes": "There are no other objects around the chair in this image. Only a very simple environment is shown, with only one object: an old wooden table and four blue chairs."
  192. Enhanced Description Output:
  193. A weathered wooden chair with a timeworn frame and soft blue cushions sits prominently at the center. The rich grain of the wood contrasts with the smooth fabric, which shows subtle signs of wear. The surroundings are minimal, featuring a matching wooden table and four identical blue chairs, creating a serene and uncomplicated environment where the chair remains the clear focal point. The warm, natural light highlights the texture of the wood and fabric.
  195. """
  196. },
  197. {"role": "user", "content": f"""
  198. Now, the "intrinsic_attributes" is "{goal_text_intrinsic}", the "extrinsic_attributes" is "{goal_text_extrinsic}", please follow the above requirements and examples to enhance this description, and directly output the enhanced description.
  199. """
  200. },
  202. ])
  204. return completion.choices[0].message.content
  208. def long_memory_localized(client, text_prompt, long_memory):
  210. completion = client.chat.completions.create(
  211. model="gpt-4o",
  212. timeout=500,
  213. messages=[
  214. {"role": "system", "content": "You are a helpful assistant."},
  215. {"role": "user", "content": """
  216. You are an LLM Agent with a specific goal: given a textual description of a navigation target (e.g., “A marble island in a kitchen.”)
  217. and a memory list containing instances of detected objects, each with a label, a 3D location (three numerical coordinates), and a confidence score,
  218. you need to determine the most suitable memory instance to fulfill the navigation request.
  220. **Your memory data structure:**
  221. The memory is a list of objects in the environment, where each object is represented as a JSON-like structure of the following form:
  222. {
  223. "label": "<string>",
  224. "loc": [<float or int>, <float or int>, <float or int>],
  225. "confidence": <float>
  226. }
  227. label: A textual label describing the object (e.g., “a tv”, “a kitchen island”).
  228. loc: A three-element array representing the coordinates of that object in the environment.
  229. confidence: A floating-point value indicating how confident the system is in identifying this object as described by label.
  231. **Your task:**
  233. 1.Understand the target description: You will be given a textual goal description of a navigation target, such as “A marble island in a kitchen.”
  234. Your first step is to interpret this description and deduce which object label from the memory best matches it semantically.
  235. For example, if the target is “A marble island in a kitchen,” and you have memory instances labeled “a kitchen island” or “a marble island,”
  236. you should identify that these instances correspond to the target description. Consider synonyms and close matches. If no exact label is found,
  237. choose the label that is most semantically similar to the target description. For instance, if the target mentions “a marble island” and the memory only has “a kitchen island,”
  238. you should still select the “a kitchen island” label as it is likely the intended object.
  240. 2.Identify the relevant instances: Once you have determined the best matching label, filter the memory list to only those instances whose label matches (or closely matches) that label.
  242. 3.Evaluate confidence and consolidate duplicates: Among these filtered instances, consider that multiple memory entries may actually represent the same object,
  243. possibly due to partial overlaps or multiple detections.
  245. - Look at their loc coordinates. If multiple instances with the same label have very close or nearly identical coordinates, treat them as the same object.
  246. - Determine which set of coordinates (if there are multiple distinct sets) is the most reliable representation of the object. Reliability is judged primarily by the highest confidence value. If multiple instances cluster together with similar locations, select the one with the highest confidence or, if confidence is similar, the one that best aligns with the object as described.
  248. 4.Select the final loc: After you have grouped instances and decided which group best represents the target object, output the coordinates (loc) of the best match.
  249. If multiple objects (>=3 items) match the description equally well, choose the three coordinates (loc) with the highest confidence.
  251. 5.Produce a final answer: Return the selected location coordinates as the final answer, (important!!) must be in the format '**Result**: (Nav Loc 1: [...], Nav Loc 2: [...], Nav Loc 3: [...])' or '**Result**: (Nav Loc: Unable to find)'.
  253. **Important details:**
  255. - Always provide reasoning internally (you may do it in hidden scratchpads if available) before giving the final result.
  256. The final user-visible answer should be concise and directly address the task.
  257. - If no objects are found that are semantically relevant to the target description, explicitly indicate that no suitable object was found.
  258. - Follow these steps for every input you receive.
  259. """
  260. },
  262. {"role": "user", "content": f"navigation target:{text_prompt}"},
  263. {"role": "user", "content": f"memory:{long_memory}"},
  265. {"role": "user", "content": "Now please start thinking one step at a time and then Briefly tell me the target location I need to go to and return as '**Result**: (Nav Loc 1: [...], Nav Loc 2: [...], Nav Loc 3: [...])' format, If there is no suitable target in memory, return as '**Result**: (Nav Loc: Unable to find)"},
  267. ])
  270. return completion.choices[0].message.content
  272. def image_to_base64(images: Image.Image, fmt="JPEG") -> str:
  274. base64_images = []
  275. for img in images:
  276. output_buffer = BytesIO()
  277. img.save(output_buffer, format=fmt)
  278. byte_data = output_buffer.getvalue()
  279. base64_str = base64.b64encode(byte_data).decode('utf-8')
  280. base64_images.append(base64_str)
  282. return base64_images
  284. # def succeed_determine(client, text_prompt, obs):
  285. # base64_image = image_to_base64(obs)
  287. # completion = client.chat.completions.create(
  288. # model="gpt-4o",
  289. # messages=[
  290. # {"role": "system", "content": "You are a helpful assistant."},
  291. # {"role": "user", "content": """
  292. # You are an AI assistant tasked with assessing whether a navigation task has been successfully completed. You will be given:
  293. # 1.A observation image (the image observed at the end of navigation).
  294. # 2.A text description of the target object the navigation aimed to locate.
  295. # Your goal is to determine:
  296. # 1.Explanation: Provide a concise explanation showing how you arrived at your conclusion, referencing any matching or non-matching details between the final observation and the target description.
  297. # 2.Success or not: Does the final observation indicate that the agent has found the target object?
  299. # **Important Requirements**
  300. # Before your success verdict, provide an explanation on a line, starting with Explanation: and then your reasoning.
  301. # If the final observation includes the target object based on the textual description, respond with exactly Success: yes.
  302. # If the final observation does not include the target object, respond with exactly Success: no.
  304. # Do not add extra lines or deviate from the required format.
  305. # **Format of Your Answer**
  306. # First line: Explanation: [Your explanation here]
  307. # Second line: Success: yes OR Success: no
  308. # """
  309. # },
  312. # {"role": "user",
  313. # "content": [
  314. # {
  315. # "type": "text",
  316. # "text": "observation image:"},
  317. # {
  318. # "type": "image_url",
  319. # "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"},},
  320. # ]
  321. # },
  322. # {"role": "user", "content": f"target description:{text_prompt}"},
  324. # {"role": "user", "content": "Now please start thinking step by step"},
  326. # ])
  329. # return completion.choices[0].message.content
  331. def succeed_determine(client, text_prompt, obss):
  332. base64_images = image_to_base64(obss)
  334. messages = [
  335. {"role": "system", "content": "You are a helpful assistant."},
  336. {"role": "user", "content": [
  338. {
  339. "type": "text", "text":
  340. """
  341. You will be provided with 2 navigation observation images from from different perspectives of the agent, and a textual description of the navigation goal. Please follow the steps below to determine whether the current navigation task is successful:
  342. 1.Determine Target Presence: Analyze the provided images one by one, to ascertain whether the navigation goal is present in these images. This means evaluating if the agent has arrived near the target location.
  343. 2. Output Format:
  344. First Line: Success: yes OR Success: no
  345. Second Line: Give your analysis results in detail
  346. Examples
  347. '''
  348. Success: yes
  349. [analysis results]
  350. '''
  351. or
  352. '''
  353. Success: no
  354. [analysis results]
  355. '''
  357. Please analyze according to the above requirements and respond strictly in the specified format.
  358. """
  359. },
  360. {
  361. "type": "text", "text": f"target description: {text_prompt}"
  362. }
  364. ]
  365. },]
  367. for idx, base64_image in enumerate(base64_images):
  368. messages[1]["content"].extend([
  369. {"type": "text", "text": f"observation view {idx}:"},
  370. {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
  371. ])
  373. messages[1]["content"].extend([
  374. {"type": "text", "text": "Now please start thinking step by step"},
  375. ])
  377. completion = client.chat.completions.create(
  378. model="gpt-4o",
  379. timeout=500,
  380. messages=messages
  381. )
  384. return completion.choices[0].message.content
  388. def succeed_determine_singleview(client, text_prompt, obss):
  389. base64_images = image_to_base64(obss)
  391. messages = [
  392. {"role": "system", "content": "You are a helpful assistant."},
  393. {"role": "user", "content": [
  395. {
  396. "type": "text", "text":
  397. """
  398. You will be provided with a navigation observation image from a robot, and a textual description of the navigation goal. Please follow the steps below to determine whether the current navigation task is successful:
  399. 1.Determine Target Presence: Analyze the provided images to ascertain whether the navigation goal is present in these images And close enough (within 2 meters). This means evaluating if the robot has arrived near the target location. Be careful not to misclassify the similar categories (e.g. sofas and chairs are easily confused)
  400. 2.Determine whether need to move forward. If you have found the target according to the step 1, you need to further determine whether you need to move forward a small step to get closer to the target object. If need to, answer 'need forward: yes'. If you think you are close enough (within 1m), answer 'need forward: no'.
  401. 3. Output Format:
  402. First Line: Success: yes OR Success: no
  403. Second Line (only when 'success: yes'): need forward: yes OR need forward: no
  404. Third Line: Give your analysis results in detail
  405. Examples
  406. '''
  407. Success: yes
  408. need forward: yes
  409. [analysis results]
  410. '''
  411. or
  412. '''
  413. Success: yes
  414. need forward: no
  415. [analysis results]
  416. '''
  417. or
  418. '''
  419. Success: no
  420. [analysis results]
  421. '''
  423. Please analyze according to the above requirements and respond strictly in the specified format.
  424. """
  425. },
  426. {
  427. "type": "text", "text": f"target description: {text_prompt}"
  428. }
  430. ]
  431. },]
  433. for idx, base64_image in enumerate(base64_images):
  434. messages[1]["content"].extend([
  435. {"type": "text", "text": f"observation view:"},
  436. {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
  437. ])
  439. messages[1]["content"].extend([
  440. {"type": "text", "text": "Now please start thinking step by step"},
  441. ])
  443. completion = client.chat.completions.create(
  444. model="gpt-4o",
  445. timeout=500,
  446. messages=messages
  447. )
  450. return completion.choices[0].message.content
  454. def succeed_determine_singleview_with_imggoal(client, img_prompt, obss):
  455. obss = [obs.resize((512, 512)) for obs in obss]
  456. img_prompt = img_prompt.resize((512, 512))
  457. base64_images = image_to_base64(obss)
  458. base64_images_goal = image_to_base64([img_prompt])
  460. messages = [
  461. {"role": "system", "content": "You are a helpful assistant."},
  462. {"role": "user", "content": [
  464. {
  465. "type": "text", "text":
  466. """
  467. You will get an image of the navigation target and an image of the current observation from the robot.
  468. 1. Analyze the main instance objects contained in the two images, especially the closest objects. Think about what objects they are? What appearance features do they have?
  469. 2. Compare the two images, combined with the analysis of the step 1, to determine whether the current robot has reached the vicinity of the target. This means that the current observation image is taken at the location of the navigation target image.
  470. 3. Determine whether need to move forward. If you think the robot has reached the target, you need to further determine whether it needs to move forward a small step to get closer to the target object.
  471. Note: The viewpoints of the two images are usually different. You need to judge carefully to avoid misjudgment.
  472. Output Format:
  473. First Line: Success: yes OR Success: no
  474. Second Line (only when 'success: yes'): need forward: yes OR need forward: no
  475. Third Line: Give your analysis results in detail
  476. Examples
  477. '''
  478. Success: yes
  479. need forward: yes
  480. [analysis results]
  481. '''
  482. or
  483. '''
  484. Success: yes
  485. need forward: no
  486. [analysis results]
  487. '''
  488. or
  489. '''
  490. Success: no
  491. [analysis results]
  492. '''
  494. Please analyze according to the above requirements and respond strictly in the specified format.
  495. """
  496. },
  497. {
  498. "type": "text", "text": f"navigation goal image:"
  499. },
  500. {
  501. "type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_images_goal[0]}"}
  502. },
  503. {
  504. "type": "text", "text": f"current observation image:"
  505. },
  506. {
  507. "type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_images[0]}"}
  508. },
  509. {
  510. "type": "text", "text": "Now please start analysing."
  511. },
  513. ],
  514. }
  515. ]
  517. completion = client.chat.completions.create(
  518. model="gpt-4o",
  519. timeout=500,
  520. messages=messages
  521. )
  524. return completion.choices[0].message.content
  528. def touching_helper(client, text_prompt, obss, model=None, processor=None):
  529. base64_image = image_to_base64(obss)[0]
  531. messages = [
  532. {"role": "user",
  533. "content":
  534. [
  535. {
  536. "type": "text", "text":
  537. """
  538. Suppose you are an agent performing a navigation task, and you need to make yourself as close to a specified target as possible. Now that you have reached the vicinity of the target, I will provide you with a description of the target object you need to reach and the current observation image. Please analyze and make decisions according to the following steps:
  539. 1. Direction judgment:
  540. Based on the observation image, is the target object already in your field of vision? If it appears, in which direction is it located? For example: straight ahead/left front/right front. If it does not appear, determine in which direction it is most likely to appear.
  541. 2. Strategy decision:
  542. Based on the direction you analyzed, if there is still a certain distance from the target, what should be your next best movement strategy? You have the following options: ['move_forward', 'turn_left', 'turn_right', 'look_up', 'look_down', 'finish_task']. Please note that you only need to consider one step of strategy.
  543. If you think you are close enough (the distance is less than 1m), you should choose the 'finish_task' strategy.
  544. 3. Output format:
  545. The final answer needs to be output in a strict format. For example: **Strategy**: 'xxx' (xxx is the strategy you choose).
  547. Please analyze according to the above requirements and respond strictly in the specified format.
  548. """
  549. },
  550. {
  551. "type": "text", "text": f"target description: {text_prompt}"
  552. },
  553. {
  554. "type": "text", "text": f"observation view:"
  555. },
  556. {
  557. "type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
  558. },
  559. {
  560. "type": "text", "text": "Now please start thinking step by step"
  561. },
  563. ],
  564. }
  565. ]
  567. # text = processor.apply_chat_template(
  568. # messages, tokenize=False, add_generation_prompt=True
  569. # )
  570. # image_inputs, video_inputs = process_vision_info(messages)
  571. # inputs = processor(
  572. # text=[text],
  573. # images=image_inputs,
  574. # videos=video_inputs,
  575. # padding=True,
  576. # return_tensors="pt",
  577. # )
  578. # inputs = inputs.to("cuda")
  580. # # Inference: Generation of the output
  581. # generated_ids = model.generate(**inputs, max_new_tokens=128)
  582. # generated_ids_trimmed = [
  583. # out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
  584. # ]
  585. # output_text = processor.batch_decode(
  586. # generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
  587. # )
  589. # return output_text[0]
  593. completion = client.chat.completions.create(
  594. model="gpt-4o",
  595. timeout=500,
  596. messages=messages
  597. )
  599. return completion.choices[0].message.content
  604. def vln_subgoal_planner_with_obs(client, text_prompt):
  606. messages = [
  607. {"role": "system", "content": "You are a helpful assistant."},
  608. {"role": "user", "content": [
  610. {
  611. "type": "text", "text":
  612. """
  613. You will get a text instruction for long-distance navigation in an indoor environment.
  614. Now, your task is to decompose the text instruction into reasonable and clear sub-task goals to help the agent complete the complex navigation task step by step. All sub-tasks need to be expressed in the form of "{move to ...}", where the target in {...} can be a word or description of an object, or a description of a room area.
  615. Next, I will give you some examples:
  617. **Text prompt:**
  618. "Walk into the hallway and at the top of the stairs turn left and walk into the bedroom. Walk past the right side of the bed and turn right into the closet and all the way through to the bathroom. Stop in front of the toilet in the bathroom."
  620. **Response:**
  621. 1. Move to the {stairs at the end of the hallway}
  622. 2. Move to the {bed in the bedroom}
  623. 3. Move to the {closet}
  624. 4. Move to the {toilet in the bathroom}
  626. **Text prompt:**
  627. "Go to the wooden stairs. Go up the stairs and go between the couch and the table. Walk into the house through the sliding glass door. Go to the television. Go to the refrigerator. Go to the front of the toaster and stop"
  629. **Response:**
  630. 1. Move to the {wooden stairs}
  631. 2. Move to the {area between a couch and a table}
  632. 3. Move to the {sliding glass door}
  633. 4. Move to the {television}
  634. 5. Move to the {refrigerator}
  635. 6. Move to the {toaster}
  637. Now, please planning the following text prompt into sub-goals and respond strictly in the specified format. Do not include any other information.
  638. """
  639. },
  640. {
  641. "type": "text", "text": f"**Text prompt:** {text_prompt}"
  642. },
  644. ],
  645. }
  646. ]
  649. completion = client.chat.completions.create(
  650. model="gpt-4o",
  651. timeout=500,
  652. messages=messages
  653. )
  656. return completion.choices[0].message.content
  660. def vln_subgoal_planner_no_object(client, text_prompt):
  662. messages = [
  663. {"role": "system", "content": "You are a helpful assistant."},
  664. {"role": "user", "content": [
  666. {
  667. "type": "text", "text":
  668. """
  669. You will get a text instruction for long-distance navigation in an indoor environment.
  670. Now, your task is to decompose the text instruction into reasonable and clear sub-task goals to help the agent complete the complex navigation task step by step.
  671. Next, I will give you some examples, the sub-task must in the format of "{...}":
  673. **Text prompt:**
  674. "Walk into the hallway and at the top of the stairs turn left and walk into the bedroom. Walk past the right side of the bed and turn right into the closet and all the way through to the bathroom. Stop in front of the toilet in the bathroom."
  676. **Response:**
  677. 1. {walk into the hallway and at the top of the stairs.}
  678. 2. {turn left and walk into the bedroom.}
  679. 3. {walk past the right side of the bed}
  680. 4. {turn right into the closet and all the way through to the bathroom.}
  681. 5. {Stop in front of the toilet in the bathroom}
  683. **Text prompt:**
  684. "Go to the wooden stairs. Go up the stairs and go between the couch and the table. Walk into the house through the sliding glass door. Go to the television. Go to the refrigerator. Go to the front of the toaster and stop"
  686. **Response:**
  687. 1. {go to the wooden stairs}
  688. 2. {go up the stairs}
  689. 3. {go between the couch and the table}
  690. 4. {walk into the house through the sliding glass door}
  691. 5. {go to the television}
  692. 6. {go to the refrigerator}
  693. 7. {go to the front of the toaster and stop}
  695. Now, please planning the following text prompt into sub-goals and respond strictly in the specified format. Do not include any other information.
  696. """
  697. },
  698. {
  699. "type": "text", "text": f"**Text prompt:** {text_prompt}"
  700. },
  703. ],
  704. }
  705. ]
  707. completion = client.chat.completions.create(
  708. model="gpt-4o",
  709. timeout=500,
  710. messages=messages
  711. )
  714. return completion.choices[0].message.content
  717. def vln_anchor_planner(client, text_prompt, obss):
  718. base64_image = image_to_base64(obss)
  720. messages = [
  721. {"role": "system", "content": "You are a helpful assistant."},
  722. {"role": "user", "content": [
  724. {
  725. "type": "text", "text":
  726. """
  727. You will act as an intelligent assistant to complete a scene navigation task. You need to analyze the provided observation images and determine the most appropriate direction of movement based on the given navigation instructions. Once the direction is determined, you need to use the corresponding image to predict and describe any important objects or environmental features that you will encounter at the destination. Your description should use natural language and provide detailed descriptions of the object's external features (texture, shape, etc.). If no obvious object is found, you should describe the surrounding environment and any significant structures or markers within it.
  729. Here are the task breakdown steps you need to follow:
  731. Direction Determination: Based on the navigation instruction, you need to choose the correct directional image from the provided images. This direction should be derived from the images provided and must be a logical response to the instruction.
  733. Object Analysis: After determining the correct directional image, analyze the image to identify what "anchor objects" you will encounter at the destination. Anchor objects represent the physical objects you will encounter when following the abstract instruction. An anchor object can be any object, such as a piece of furniture, a notable landmark, or any other clearly identifiable structure. You should provide a detailed natural language description of the anchor object, including its appearance, such as shape, color, texture, and any other notable features. If no obvious object is found in the selected direction, try to describe the appearance of the destination. This might include the terrain, nearby notable structures, or any significant features.
  735. Output Response: Finally, you need to return your analysis results and provide the anchor object description.
  737. Here is an example. You must strictly follow the format of the following response:
  739. Instruction: "Walk straight through the bathroom and exit."
  741. Your Response should be in the following format:
  743. Analysis: (your analysis here)
  745. Anchor Object: e.g., A large marble sink with gold-trimmed faucets. The sink is rectangular, with polished white marble and faint gray veining running across its surface. Above the sink, there's a framed mirror with an ornate golden frame.
  747. Now, please start your analysis based on the following input information:
  748. """
  749. },
  750. {
  751. "type": "text", "text": f"**Instruction:** {text_prompt}"
  752. },
  753. {
  754. "type": "text", "text": f"**Observation images:**"
  755. }
  757. ],
  758. }
  759. ]
  761. for idx, base64_image in enumerate(base64_image):
  762. messages[1]["content"].extend([
  763. {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
  764. ])
  766. completion = client.chat.completions.create(
  767. model="gpt-4o",
  768. timeout=500,
  769. messages=messages
  770. )
  773. return completion.choices[0].message.content
  779. def vln_anchor_planner_v2(client, text_prompt, obss):
  780. base64_image = image_to_base64(obss)
  782. messages = [
  783. {"role": "system", "content": "You are a helpful assistant."},
  784. {"role": "user", "content": [
  786. {
  787. "type": "text", "text":
  788. """
  789. You will act as an agent to complete the task of assisting scene navigation. I will provide you with a navigation instruction and a series of current observations. The navigation instruction requires the agent to move from the current position to an "object" not far away. However, the current description of the object is rough and is only a simple category description.
  791. Now, your task is to analyze the current observation and then describe the characteristics of the target object in as much detail as possible, including fine-grained elements such as appearance, shape, and texture. You may encounter two situations:
  793. 1. The target object is within the field of view of the observed image, which indicates that you can make a fine-grained description based on the observation content.
  795. 2. The target object does not exist in the field of view of the observed image. At this time, you need to associate and infer the possible characteristics of the target object as much as possible based on the indoor environment and surrounding information in the observed image, and give a description.
  797. Here is an example. You need to output the description directly without any other analysis or additional responses:
  799. Input Instruction:
  801. "Move to the marble sink in the bathroom."
  803. Your Response should like:
  805. "A large marble sink with gold-trimmed faucets. The sink is rectangular, with polished white marble and faint gray veining running across its surface. Above the sink, there's a framed mirror with an ornate golden frame."
  807. Now, please start your analysis based on the following input information:
  808. """
  809. },
  810. {
  811. "type": "text", "text": f"**Instruction:** {text_prompt}"
  812. },
  813. {
  814. "type": "text", "text": f"**Observation images:**"
  815. }
  817. ],
  818. }
  819. ]
  821. for idx, base64_image in enumerate(base64_image):
  822. messages[1]["content"].extend([
  823. {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
  824. ])
  826. completion = client.chat.completions.create(
  827. model="o3",
  828. timeout=500,
  829. messages=messages
  830. )
  833. return completion.choices[0].message.content
  837. def EQA_generate_anchor_object(client, text_prompt):
  838. messages = [
  839. {"role": "system", "content": "You are a helpful assistant."},
  840. {"role": "user", "content": [
  842. {
  843. "type": "text", "text":
  844. """
  845. You will act as an agent to complete the task of assisting scene embodied question answering. I will provide you with a question about the scene. In order to answer this question, we must first navigate to the vicinity of the instance involved in the question.
  847. Now, your task is to analyze and determine the description of the target instance you need to move to based on the current question, which can include some necessary spatial context, such as what type of room this target instance is in and what clear objects exist around it. If you think it is difficult to infer the exact target instance for the type of question provided, please output "We need to go around and check"
  849. I will provide you with some examples. You need to output the description directly without including other analysis and additional response content.
  851. Example 1:
  853. Question: What is the white object on the wall above the TV?
  855. Response: Now, we need to go to {A TV mounted on the wall.}
  857. Example 2:
  859. Question: What is in between the two picture frames on the blue wall in the living room?
  861. Response: Now, we need to go to {A blue wall in the living room with two picture frames on it.}
  863. Example 3:
  865. Question: What should I do to cool down?
  867. Response: We need to go around and check.
  869. Your Turn:
  870. """
  871. },
  872. {
  873. "type": "text", "text": f"Question: {text_prompt} Response:"
  874. }
  876. ],
  877. }
  878. ]
  881. completion = client.chat.completions.create(
  882. model="o3-mini",
  883. timeout=500,
  884. messages=messages
  885. )
  888. return completion.choices[0].message.content
  891. def EQA_Answer_o3(client, text_prompt, obss):
  892. base64_image = image_to_base64(obss)
  894. messages = [
  895. {"role": "system", "content": "You are a helpful assistant."},
  896. {"role": "user", "content": [
  898. {
  899. "type": "text", "text":
  900. f"""
  901. You are an intelligent question-answering robot. I will ask you some questions about indoor spaces, and you must give answers.
  903. You will see a set of images collected from the same location. The location of these images is related to the question, and you must carefully reason from them to get the correct answer. If you think there is not enough information to get the answer at that location, please try to guess a close and most likely answer.
  905. Based on the user query, you must output "text" to answer the question asked by the user. Here are some examples:
  907. Example1:
  908. Question: What is to the left of the mirror?
  909. Answer: A plant in a tall vase.
  911. Example2:
  912. Question: Where is the mirror?
  913. Answer: Next to the staircase above the dark brown cabinet.
  915. Your Turn:
  916. Question: {text_prompt}
  918. """
  919. },
  920. {
  921. "type": "text", "text": f"**Observation images:**"
  922. }
  924. ],
  925. }
  926. ]
  928. for idx, base64_image in enumerate(base64_image):
  929. messages[1]["content"].extend([
  930. {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
  931. ])
  933. completion = client.chat.completions.create(
  934. model="o3-mini",
  935. timeout=500,
  936. messages=messages
  937. )
  940. return completion.choices[0].message.content
  942. def EQA_Answer_4o(client, text_prompt, obss):
  943. base64_image = image_to_base64(obss)
  945. messages = [
  946. {"role": "system", "content": "You are a helpful assistant."},
  947. {"role": "user", "content": [
  949. {
  950. "type": "text", "text":
  951. f"""
  952. You are an intelligent question-answering robot. I will ask you some questions about indoor spaces, and you must give answers.
  954. You will see a set of images collected from the same space. These observation images is related to the question, and you must carefully reason from them to get the correct answer. Please do not respond with "I can't answer that because I didn't see..." If you think the images are not accurate enough to get an answer, please try to guess the most likely answer.
  956. Based on the user query, you must output "text" to answer the question asked by the user. Here are some examples:
  958. Example1:
  959. Question: What is to the left of the mirror?
  960. Answer: A plant in a tall vase.
  962. Example2:
  963. Question: Where is the mirror?
  964. Answer: Next to the staircase above the dark brown cabinet.
  966. Your Turn:
  967. Question: {text_prompt}
  969. """
  970. },
  971. {
  972. "type": "text", "text": f"**Observation images:**"
  973. }
  975. ],
  976. }
  977. ]
  979. for idx, base64_image in enumerate(base64_image):
  980. messages[1]["content"].extend([
  981. {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
  982. ])
  984. completion = client.chat.completions.create(
  985. model="gpt-4o",
  986. timeout=500,
  987. messages=messages
  988. )
  991. return completion.choices[0].message.content

LLMAgent.py at commit ad4c544, no license · at the source

Overview

Authors: Shouwei Ruan1,2, Liyuan Wang1,3, Caixin Kang2, Qihui Zhu2, Songming Liu1, Xingxing Wei2, Hang Su1
ORCID iDs: Shouwei Ruan, Liyuan Wang
  1. Department of Computer Science and Technology, Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University, Beijing, China
  2. Institute of Artificial Intelligence, Beihang University, Beijing, China
  3. Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China
Institutions: Beihang University (China); Tsinghua University (China)
Journal: Nature communications, volume 17, issue 1, article 8061
Dates: received 13 October 2025; accepted 2 June 2026; published online 27 June 2026
Type: Research article · Language: English
License: CC BY-NC-ND
Identifiers: DOI 10.1038/s41467-026-74358-5 · PMID 42364983 · PMCID PMC13454140 · OpenAlex W7166300381
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: human (organism), cognitive (subfield)
Methods: Graphs, Machine learning
Keywords: Computer science, Cognitive neuroscience
MeSH: Artificial Intelligence*, Brain*, Spatial Memory*, Spatial Navigation*, Cognition, Cues, Humans, Large Language Models (* major topic)
Topic: EEG and Brain-Computer Interfaces (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Citations: not cited yet (Europe PMC); 87 references in the paper

Abstract

The abstract is not reproduced here: the paper's license (CC BY-NC-ND) does not allow it. Read it in the paper, at the publisher or on Europe PMC.

Repositories

Its files are read in the Code ↔ Paper reader above, with 16 matches between paragraphs and lines of code.

facebookresearch/habitat-sim

License: MIT
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 57ee4941dc4765240f0f91f70b2c97a919bf9038, 7 May 2026
Languages: C/C++ (196), C++ (191), Python (124), Shell (11), Jupyter (8), C (1), CUDA (1)
Size: 930 files, 532 scripts
Software Heritage: archived
Found in: “Data availability”
Holds: README, license file, environment (pyproject.toml, requirements.txt, setup.cfg, setup.py, conda-build/Dockerfile, conda-build/habitat-sim-mutex/conda_build_config.yaml), tests, continuous integration, documentation, 8 notebooks
Not found: CITATION.cff
Tools: NumPy (62 files), Pillow (11 files), Matplotlib (8 files), PyTorch (8 files), imageio (3 files), Numba (2 files), OpenCV (1 file), SciPy (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
534 files
At the source: github.com/facebookresearch/habitat-sim/blob/main/DATASETS.md

facebookresearch/habitat-lab

License: MIT
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 0fb6f43ffe806a8088a171b036336c093bcf604e, 7 May 2026
Languages: Python (418), Jupyter (7), Shell (6)
Size: 721 files, 431 scripts
Software Heritage: archived
Found in: “Data availability”
Holds: README, license file, environment (Dockerfile, pyproject.toml, setup.cfg, habitat-baselines/setup.py, habitat-hitl/requirements.txt, habitat-hitl/setup.py, habitat-lab/requirements.txt, habitat-lab/setup.py, scripts/habitat_dataset_processing/requirements.txt, scripts/habitat_dataset_processing/setup.py, habitat-baselines/habitat_baselines/il/requirements.txt, habitat-baselines/habitat_baselines/rl/requirements.txt), tests, continuous integration, documentation, 7 notebooks
Not found: CITATION.cff
Tools: NumPy (158 files), PyTorch (76 files), OpenCV (10 files), Matplotlib (8 files), Pillow (8 files), imageio (6 files), Numba (2 files), SciPy (2 files)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
433 files
At the source: github.com/facebookresearch/habitat-lab/blob/main/DATASETS.md

naokiyokoyama/ovon

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 8300fcc9fcd820637ac202cb43db080229f53410, 15 May 2025
Languages: Python (82), Shell (18)
Size: 126 files, 100 scripts
Software Heritage: not archived
Found in: “Data availability”
Holds: README, environment (pyproject.toml, setup.py), documentation
Not found: license file, CITATION.cff, tests, continuous integration
Tools: PyTorch (41 files), NumPy (32 files), Matplotlib (6 files), seaborn (6 files), pandas (5 files), OpenCV (3 files), scikit-learn (3 files), Pillow (2 files), Hugging Face Transformers (2 files), SciPy (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
101 files
At the source: github.com/naokiyokoyama/ovon

XinyuSun/PSL-InstanceNav

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 7b9faa6c382c50d2920ebd129b3376ca8fbd9ea4, 6 June 2025
Languages: Python (18), Shell (2)
Size: 29 files, 20 scripts
Software Heritage: not archived
Found in: “Data availability”
Holds: README, environment (requirements.txt, setup.cfg, setup.py)
Not found: license file, CITATION.cff, tests, continuous integration, documentation
Tools: NumPy (10 files), PyTorch (9 files), OpenCV (2 files), Pillow (2 files)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
21 files
At the source: github.com/XinyuSun/PSL-InstanceNav

facebookresearch/open-eqa

License: MIT
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: cfa3fce4595c1622bb2f8a38ae2ca9aae9eb685b, 20 September 2024
Languages: Python (25), Jupyter (1)
Size: 49 files, 26 scripts
Software Heritage: not archived
Found in: “Data availability”
Holds: README, license file, environment (requirements.txt, setup.py), 1 notebook
Not found: CITATION.cff, tests, continuous integration, documentation
Tools: NumPy (8 files), OpenCV (5 files), Pillow (4 files), imageio (2 files), Matplotlib (1 file), PyTorch (1 file), Hugging Face Transformers (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
28 files
At the source: github.com/facebookresearch/open-eqa

Heathcliff-saku/BSC-Nav

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: ad4c544f7d1feb980ffad33d4eefa8ad2e39fad0, 6 November 2025
Languages: Python (466), Shell (15), Jupyter (7)
Size: 862 files, 488 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, environment (environment.yml, requirements.txt, third-party/habitat-lab/Dockerfile, third-party/habitat-lab/pyproject.toml, third-party/habitat-lab/setup.cfg, third-party/habitat-lab/habitat-baselines/setup.py, third-party/habitat-lab/habitat-hitl/setup.py, third-party/habitat-lab/habitat-lab/setup.py), tests, documentation, 7 notebooks
Not found: license file, CITATION.cff, continuous integration
Tools: NumPy (195 files), PyTorch (103 files), Pillow (26 files), OpenCV (24 files), imageio (18 files), SciPy (17 files), Matplotlib (15 files), scikit-learn (15 files), Hugging Face Transformers (10 files), h5py (2 files), NetworkX (2 files), Numba (2 files), pandas (1 file), TensorFlow (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
489 files
At the source: github.com/Heathcliff-saku/BSC-Nav

Code availability statement

The paper has a code availability statement. Its license (CC BY-NC-ND) does not allow reproducing it here; in short, from what the harvester recognized in it:

Read it in the paper: doi.org/10.1038/s41467-026-74358-5.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 6 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 1,597 scripts, each with its path and the digest of its content;
  • 16 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Code and data availability statement

The paper has a code and data availability statement. Its license (CC BY-NC-ND) does not allow reproducing it here; in short, from what the harvester recognized in it:

Read it in the paper: doi.org/10.1038/s41467-026-74358-5.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 7 authors, 2 keywords, 8 MeSH terms, 27 references.

Cite

This paper

Ruan, S., Wang, L., Kang, C., Zhu, Q., Liu, S., Wei, X., & Su, H. (2026). Brain-inspired spatial intelligence for embodied agents. Nature communications, 17(1), 8061. https://doi.org/10.1038/s41467-026-74358-5

BibTeX

@article{ruan2026brain,
author = {Ruan, Shouwei and Wang, Liyuan and Kang, Caixin and Zhu, Qihui and Liu, Songming and Wei, Xingxing and Su, Hang},
title = {{Brain-inspired spatial intelligence for embodied agents}},
journal = {Nature communications},
year = {2026},
month = jun,
volume = {17},
number = {1},
pages = {8061},
publisher = {Nature Publishing Group},
issn = {2041-1723},
doi = {10.1038/s41467-026-74358-5},
url = {https://doi.org/10.1038/s41467-026-74358-5},
pmid = {42364983},
pmcid = {PMC13454140}
}

RIS

TY - JOUR
AU - Ruan, Shouwei
AU - Wang, Liyuan
AU - Kang, Caixin
AU - Zhu, Qihui
AU - Liu, Songming
AU - Wei, Xingxing
AU - Su, Hang
TI - Brain-inspired spatial intelligence for embodied agents
T2 - Nature communications
J2 - Nat Commun
PY - 2026
DA - 2026/06/27
VL - 17
IS - 1
SP - 8061
SN - 2041-1723
PB - Nature Publishing Group
DO - 10.1038/s41467-026-74358-5
UR - https://doi.org/10.1038/s41467-026-74358-5
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s41467-026-74358-5",
"type": "article-journal",
"title": "Brain-inspired spatial intelligence for embodied agents",
"container-title": "Nature communications",
"author": [
{
"family": "Ruan",
"given": "Shouwei"
},
{
"family": "Wang",
"given": "Liyuan"
},
{
"family": "Kang",
"given": "Caixin"
},
{
"family": "Zhu",
"given": "Qihui"
},
{
"family": "Liu",
"given": "Songming"
},
{
"family": "Wei",
"given": "Xingxing"
},
{
"family": "Su",
"given": "Hang"
}
],
"container-title-short": "Nat Commun",
"volume": "17",
"issue": "1",
"page": "8061",
"DOI": "10.1038/s41467-026-74358-5",
"PMID": "42364983",
"PMCID": "PMC13454140",
"ISSN": "2041-1723",
"publisher": "Nature Publishing Group",
"URL": "https://doi.org/10.1038/s41467-026-74358-5",
"language": "en",
"issued": {
"date-parts": [
[
2026,
6,
27
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41598-026-43529-1 [code]
A spiking neural network inspired by neuroscience and psychology for Western mode- and key-conditioned music learning and composition.
Journal: Scientific reports
In common: imageio, TensorFlow, NetworkX, 9 other tools, cognitive, 1 reference
[2] doi:10.1016/j.xcrm.2026.102766 [code]
A longitudinal single-cell and spatial multiomic atlas of pediatric high-grade glioma.
Journal: Cell reports. Medicine
In common: imageio, Numba, TensorFlow, 10 other tools
[3] doi:10.1371/journal.pcbi.1014571 [code]
SynAPSeg: A novel dataset and image analysis framework for deep learning-based synapse detection and quantification.
Journal: PLoS computational biology
In common: imageio, Numba, TensorFlow, 10 other tools
[4] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: Hugging Face Transformers, Numba, TensorFlow, 10 other tools
[5] doi:10.1038/s41598-026-57519-w [code]
Automated segmentation of neurons and spinal cord structures in immunofluorescence images using SpineDL.
Journal: Scientific reports
In common: imageio, Numba, NetworkX, 9 other tools
[6] doi: [code]
Real-time closed-loop feedback system for mouse mesoscale cortical signal and movement control
Journal: eLife
In common: imageio, Numba, TensorFlow, 9 other tools
[7] doi:10.1364/boe.605322 [code]
Generalized plaque digitization framework for multi-dimensional mesoscopic images.
Journal: Biomedical optics express
In common: imageio, TensorFlow, OpenCV, 9 other tools
[8] doi:10.7554/elife.110588 [code]
Opening the black box toward a modular approach to spike sorting.
Journal: eLife
In common: Numba, TensorFlow, NetworkX, 9 other tools
[9] doi:10.1038/s41467-026-74357-6 [code]
Hippocampo-neocortical interaction as compressive retrieval-augmented generation.
Journal: Nature communications
In common: Hugging Face Transformers, NetworkX, Pillow, 7 other tools, 2 references
[10] doi:10.3389/fnsys.2026.1822122 [code]
Convergence-divergence circuits for multimodal integration of innate and learned opponent valences.
Journal: Frontiers in systems neuroscience
In common: Numba, TensorFlow, NetworkX, 9 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.