问HN：你们如何防止AI代理在生产环境中失控？
大家好！<p>似乎有一个持续的趋势（以及我的直觉）是，许多公司正在从聊天机器人转向能够实际执行操作的人工智能代理——例如调用API、修改数据库、进行购买等。我很好奇：如果你们在生产环境中使用这些代理，除了防止提示注入的安全层之外，你们是如何处理的？<p>问题：<p>- 有什么措施可以阻止你的代理执行意外操作（例如删除记录、未经授权的交易）？
- 你们是否遇到过代理失控的情况，导致损失了资金或数据？
- 目前的工具（身份和访问管理政策、审批工作流程、监控）是否足够，还是存在漏洞？<p>我想弄清楚这是否是一个值得解决的实际问题，或者现有的方法是否已经足够有效。
查看原文
Hi all!<p>There seems to be an ongoing trend (and my gut feeling) of companies moving from chatbots to AI agents that can actually execute actions—calling APIs, modifying databases, making purchases, etc.
I'm curious: if you're running these in production, how are you handling the security layer beyond prompt injection defenses?<p>Questions:<p>- What stops your agent from executing unintended actions (deleting records, unauthorized transactions)?
- Have you actually encountered a situation where an agent went rogue, and you lost money or data?
- Are current tools (IAM policies, approval workflows, monitoring) enough, or is there a gap?<p>Trying to figure out if this is a real problem worth solving or if existing approaches are working fine.