The Challenge: Unlocking Data Potential
- Provide a highly interactive and customizable dashboarding experience.
- Support a diverse range of data visualization widgets.
- Incorporate AI-driven features like natural language querying and intelligent suggestions.
- Ensure scalability, reliability, and high performance.
- Streamline data pipeline and reporting processes.
My Role & Contributions: From Vision to Reality
- Full-Stack Architecture & Development: I was instrumental in defining the overall architecture.
- Frontend: Developed a highly responsive and dynamic user interface using Next.js and TypeScript, focusing on a best-in-class user experience for creating and interacting with visualizations.
- Backend (AI & Core Logic): Designed and implemented the Python-based backend for the AI layer, integrating LangChain and OpenAI and Gemini models to power natural language querying (allowing users to ask questions in plain English to generate charts) and intelligent data-driven suggestions. For other backend components and CRM integrations, I also worked with Java and Spring Boot.
- Database Interaction: Designed schemas and managed data flows using MySQL and PostgreSQL, with optimized querying via JDBC and Spark SQL for performance.
- The "Visualizer" Core: A central piece was the "Visualizer" component, which I architected to support a wide array of dynamic widgets. This involved:
- Implementing diverse chart types: KPIs, Bar, Line, Area, Scatter, Packed Bubble, Pie, Donut, Funnel, Tree maps, Box plots.
- Developing advanced Table and Pivot Table functionalities, including a hierarchical Tree View and conditional formatting.
- Enabling features like interactive drill-downs, widget customization, and the ability for users to view underlying SQL queries.
- AI Integration: This was a critical aspect. I leveraged OpenAI models to:
- Allow users to generate charts and insights using natural language queries.
- Provide intelligent suggestions based on the data patterns.
- This resulted in a 90% reduction in manual effort for users creating visualizations.
- Team Leadership & Mentorship: I led and mentored a team of engineers, fostering a culture of innovation and guiding them through the design, development, and deployment phases.
- Performance & Scalability: I benchmarked critical components, such as rendering hundered's of charts/visualizations powering from different data sources with millions of records.
Data Points per Chart | Avg. Render Time (ms) | P95 Render Time (ms) |
|---|---|---|
| 10,000 | 120ms | 180ms |
| 50,000 | 450ms | 600ms |
| 100,000 | 980ms | 1300ms |
- Documentation: I played a part in creating comprehensive documentation, similar to our internal "Data Visualization Guide," ensuring clarity for both users and fellow developers.
This project was a key factor in my receiving the "Employee of the Year" award in 2024 at InnCreTech, primarily for its significant contribution to acquiring new customers and enhancing our product offering.
Key Features of the Platform I Developed
- Dynamic Dashboarding: Users can create, customize, and share interactive dashboards.
- Comprehensive Widget Library: Including KPIs, Bar/Line/Area charts, Scatter/Bubble plots, Pie/Donut charts, Funnel/Tree maps, Box plots, and advanced Tables/Pivot Tables.
- AI-Powered Insights: Natural Language Querying and intelligent suggestions.
- Interactive Data Exploration: Drill-downs (widget, column, journey), advanced filtering, and data linking across workbooks.
- Embedding Capabilities: Allowing analytics to be integrated into third-party applications.
- Robust Configuration: Extensive options for configuring and setting widgets, including conditional formatting and formula creation.
Technologies Leveraged
- Frontend: Next.js, React.js, TypeScript, (Potentially HighCharts or similar for charting)
- Backend: Python, LangChain, OpenAI, Gemini, Java, Spring Boot
- Databases: MySQL, PostgreSQL
- Data Querying: JDBC, Spark SQL
- DevOps/Cloud (Exposure/Deployment): GCP, Docker, Kubernetes