Lilly Tech Systems
Best Practices Intermediate
Building a trusted AI chatbot for network operations requires careful attention to security, access control, conversation quality, and continuous improvement based on user feedback.
Security and Access Control
Principle of Least Privilege: The chatbot should have the minimum access needed for its function. Use dedicated service accounts with read-only access. Implement RBAC so different user roles get different capabilities — junior engineers can query, senior engineers can execute approved remediation actions.
Conversation Design
| Principle | Implementation |
|---|---|
| Be transparent | Always indicate when the chatbot is querying live systems vs. knowledge base |
| Show confidence | Indicate certainty level in diagnoses; say "I'm not sure" when appropriate |
| Provide sources | Link to runbooks, monitoring dashboards, or documentation used in the answer |
| Know limitations | Clearly state when an issue requires human expertise and suggest escalation |
Monitoring and Improvement
- Track usage metrics
Monitor query volume, response time, escalation rate, and user satisfaction scores.
- Review conversations
Regularly review chatbot conversations to identify gaps in knowledge and incorrect responses.
- Update knowledge base
Add new runbooks and incident resolutions as they occur. Remove outdated documentation.
- Gather feedback
Use reaction-based feedback (thumbs up/down) to measure answer quality and prioritize improvements.
Common Pitfalls
- Over-promising capabilities — Set clear expectations about what the chatbot can and cannot do
- Stale knowledge — Outdated runbooks lead to incorrect advice; automate knowledge refresh
- Ignoring context — Ensure the chatbot considers recent changes and ongoing incidents
- No escalation path — Always provide a way to reach a human engineer
Course Complete!
You now have the knowledge to build and deploy an AI chatbot for network operations. Start with a simple knowledge-retrieval bot, add live diagnostics, and deploy to your team's Slack or Teams workspace.
← Back to Course Overview